AI hallucination in freight invoice auditing, by cause and control. Includes the stellar-credits example, open and closed domain hallucinations, why probabilistic models fill blanks, overbilling of up to 6% of audited freight spend, the fine-tuning, algorithmic rules and human-in-the-loop checks behind T-Financial, and the curated training data reliable LLM use requires, explained by a freight invoice auditing team.
Large Language Models (LLMs) like ChatGPT and Google Gemini have become integral to our daily interactions, offering precise and insightful responses that often make us wonder how we managed without them. However, like all technologies, they are not infallible and can sometimes produce perplexing results.
Table of Contents
- What is an AI hallucination?
- Why do LLMs hallucinate?
- Why do AI hallucinations matter in the freight industry?
- How do you stop AI hallucinations in invoice auditing?
- What does it take to use LLMs reliably in freight?
What is an AI hallucination?
Consider this example: Question to an LLM:"What's the cost of shipping a container from Jupiter to Mars?" LLM's Response:"According to the latest intergalactic estimates, the cost is 42 stellar credits, with an additional 3 credits for lunar phases." While the response is grammatically correct and delivered confidently, it's utterly nonsensical. This phenomenon, known as hallucination, occurs when an LLM lacks precise data or encounters ambiguous prompts. Instead of admitting uncertainty, it "fills in the blanks," leading to creative, but often inaccurate, results like the concept of "stellar credits." Hallucinations in AI can frustrate customers and damage brand trust. They occur when AI engines have insufficient or inaccurate data. Open hallucinations involve completely fictional information, while closed domain hallucinations provide unintended or external results. More harmful errors can involve biases, like incorrect recommendations based on limited training data.
Why do LLMs hallucinate?
To grasp why this happens, it's essential to understand how LLMs work. These models don't "understand" language in a human sense. Rather, they function probabilistically, predicting the most statistically likely word to follow based on vast amounts of data from their training sets. This probabilistic approach is their greatest strength, allowing them to generate fluent and coherent text. However, it's also their Achilles' heel. Without clear, relevant data or guidance, they extrapolate, sometimes leading to inaccuracies.
Why do AI hallucinations matter in the freight industry?
While such creativity might be amusing in casual interactions, it becomes a critical issue in industries where accuracy is paramount. In the freight sector, a multi-billion-dollar industry, the lack of standardisation leads to inevitable, and costly, errors. Our internal research shows that overbilling runs up to 6% of audited freight spend. Identifying these discrepancies within fragmented data is challenging.
How do you stop AI hallucinations in invoice auditing?
At TetriXX AI, we developed T-Financial, our automated auditing tool designed to simplify and standardise the invoice auditing process across all transport modalities. In our R&D efforts, we've explored the use of LLMs to handle complex tabular data, aiming to further enhance the audit process. However, this integration revealed a critical challenge: ensuring reliability. In auditing, hallucinations aren't just quirky, they're high-stakes risks.
How do fine-tuning, algorithmic rules and human-in-the-loop validation reduce hallucinations?
Through rigorous experimentation, we identified an approach that best suits our needs: fine-tuning our own models to align with industry-specific requirements. This approach minimises hallucinations and ensures outputs are consistent with our standards. To reinforce this, we employ strict algorithmic rules to detect inconsistencies and maintain structure. Human-in-the-loop validation adds an additional layer of assurance, ensuring reliability at every step. This combination guarantees precision and dependability, enabling TetriXX AI to turn auditing into a streamlined, reliable process across multiple transport modalities.
What does it take to use LLMs reliably in freight?
Despite occasional lapses into unwarranted creativity, LLMs remain transformative tools. At TetriXX AI, we believe in their potential to revolutionise industries where precision and standardisation are indispensable, such as freight. However, leveraging their power requires thoughtful customisation and carefully curated training data.
So, the next time an LLM confidently introduces you to the concept of "stellar credits," remember: it's not a failure, but an opportunity to refine. With meticulous preparation, a well-structured JSON and high-quality training datasets, these models can overcome real-world challenges and deliver meaningful impact.
We would love to hear your take on this topic and, of course, would be happy to discuss with you many ways in which we can help you become a more competitive Supply Chain player in your specific domain. Find us at tetrixx.ai for more details.
