AI Hallucination in Engineering: Why It Happens and How to Catch It

Last updated: August 2026

The dangerous failure of an AI tool in engineering is not the answer it refuses to give. It is the one it gives with total confidence that happens to be wrong: a yield strength that looks right but is not, a citation to an “ASCE 7 section” that does not exist, a load factor pulled from nowhere. This is AI hallucination in engineering, and because the output is fluent and plausible, it slips past a tired reviewer far more easily than an obvious error would. This guide explains why language models fabricate, exactly how it shows up in engineering work, where it does the most damage, and how to catch it.

Hallucination is one of the failure modes we cover in our broader look at the limitations of AI in engineering, and it sits alongside the wider question of whether AI can do engineering calculations at all. Here we go deep on this one problem, because for an engineer it is the one most likely to reach a drawing. For the tools themselves, see our roundup of AI tools for mechanical engineers.

What an AI hallucination actually is

A large language model generates text by predicting the next most likely token from patterns in its training data. It has no internal model of physics, no copy of the code book, and no sense of whether a statement is true, only of whether it reads like the kind of thing that follows. When the honest answer is “I do not know,” the fluent, plausible-sounding continuation is often a confident fabrication instead. A 2025 paper from OpenAI and Georgia Tech researchers argues this is structural: hallucinations arise from natural statistical pressure during training, and are reinforced because common benchmarks reward a confident guess over admitting uncertainty (Kalai et al., 2025).

The reasoning is shallower than it looks, too. In a controlled study, adding a single clause that seemed relevant but changed nothing caused accuracy to drop by up to 65 percent across state-of-the-art models, which led the authors to conclude the models replicate reasoning steps from their training data rather than performing genuine logical reasoning (Mirzadeh et al., 2024). For an engineer that is the whole point: a tool that pattern-matches its way to an answer can produce one that is dimensionally plausible and numerically wrong.

How hallucination shows up in engineering

Generic explainers stop at the mechanism. What matters for practice is the specific shapes it takes on engineering deliverables.

• Fabricated material properties. Ask for a yield strength, elastic modulus, or fatigue limit and a model will often supply a confident number that is close enough to look right and wrong enough to matter. It has no datasheet; it is estimating.
• Invented code and standard clauses. A model will cite “ASCE 7 Section 2.7” or a specific NEC article with total assurance even when that clause number does not exist or does not say what it claims. The citation format is learned; the content is not verified.
• References and datasheets that do not exist. Papers, textbook page numbers, part numbers, and manufacturer datasheets can all be fabricated wholesale, complete with plausible authors and titles.
• Plausible-but-wrong equations and factors. A load combination, a safety factor, or a formula can arrive slightly altered from the real one, which is harder to spot than a nonsense answer.
• Unit and conversion errors. Models routinely slip between metric and imperial or drop a factor; that specific problem has its own well-known pattern, but it belongs on this list.

Are newer models getting better?

Yes, substantially, and that is worth saying plainly rather than fear-mongering. When researchers checked the citations ChatGPT generated for literature reviews, 55 percent of GPT-3.5’s citations were fabricated against 18 percent for GPT-4; among the citations that were real, 43 percent from GPT-3.5 and 24 percent from GPT-4 still contained substantive errors (Walters and Wilder, 2023). The newer model was a major improvement, roughly a threefold drop in outright fabrication.

But “better” is not “solved.” Even the stronger model fabricated almost one citation in five and mis-stated roughly one real citation in four, and the reasoning fragility measured in the GSM-Symbolic work above was found in late-2024 and 2025 state-of-the-art models. Treat the trend as real and the risk as still present. Every rate above is a measurement of a specific model generation, not a guarantee about the one you are using today.

Where hallucination does the most damage

Not every hallucination matters equally. A wrong fact in a brainstorming session is cheap; the same wrong fact in a sealed calculation is not. The high-consequence zones are the ones where an unverified number flows straight into something binding: a stamped or sealed design, a code-compliance determination, or a procurement decision where a fabricated part number or datasheet leads to the wrong component being ordered. Those are exactly the outputs that carry legal and safety weight, and they are precisely where a plausible-but-wrong value is most likely to survive because everything around it looks professional. The lower-risk uses, drafting explanatory text, restructuring notes, or getting an orientation to an unfamiliar topic you will verify anyway, are where AI earns its place.

How to catch and prevent it

The defense is a short, repeatable discipline, not a leap of faith.

• Treat every AI-quoted number as unverified until checked against a primary source: the manufacturer datasheet, the mill certificate, the actual code text, or a trusted handbook. Since even the better model fabricated about one reference in five, both questions matter: does the source exist, and does it say exactly what was claimed?
• Ask for sources, then open them. Fabricated clause numbers and papers look right on the page; their existence is the first gate, and it is a fast one to check.
• Use a real computational engine for the math. A verified solver, a spreadsheet with checked formulas, or a tool like Wolfram Alpha will not pattern-match its way to a wrong sum the way a chatbot can. Do not trust inline mental arithmetic for anything that matters.
• Know that grounding helps but does not cure. Retrieval and citation features reduce fabrication, but the incentive to answer confidently is structural, so verification stays your job.
• Keep a licensed engineer in the loop. Human review remains the backstop for anything feeding a stamped design, a compliance call, or a purchase. AI is an assistant, not the engineer of record.

Stated as one rule: never accept an AI-quoted value, standard clause, or part number without independent verification against the primary source.

Frequently asked questions

Why does AI make up material properties or code clauses?

Because a language model predicts plausible text rather than retrieving verified facts. It has no datasheet and no copy of the code book, so when you ask for a yield strength or a clause number it produces the kind of answer that usually follows, which can be confidently wrong. Researchers describe this as a structural result of how models are trained and evaluated, since benchmarks tend to reward a confident guess over admitting uncertainty. Always confirm the value against the primary source.

Can ChatGPT cite fake engineering references or standards?

Yes. In a 2023 study of ChatGPT-generated citations, 55 percent of GPT-3.5’s citations and 18 percent of GPT-4’s did not exist at all, and many of the real ones still contained errors. Invented paper titles, textbook page numbers, part numbers, and standard clause numbers are a common failure mode. Check that any cited source exists before you rely on what it supposedly says.

Are AI hallucinations getting better with newer models?

They are getting less frequent. The same 2023 study found GPT-4 fabricated citations at roughly a third of GPT-3.5’s rate, a clear improvement. But the failure mode is not eliminated: the stronger model still fabricated nearly one citation in five, and 2024 to 2025 research shows the underlying reasoning remains fragile. Treat the improvement as real and the risk as still present, and verify regardless of which model you use.

How do I check whether an AI-generated calculation is correct?

Rework the arithmetic in a verified engine such as a spreadsheet with checked formulas or a computational tool, rather than trusting the chatbot’s inline math. Confirm every input value against a primary source, check units and dimensions explicitly, and verify that any equation or factor matches the actual standard. For anything that will be stamped, a licensed engineer should review the result. If the tool cannot show a source for a number, treat the number as unverified.

What is the difference between an AI error and a hallucination?

An ordinary error is usually visible: a nonsensical answer, a crash, a refusal. A hallucination is a confident, fluent, plausible output that happens to be fabricated, which is what makes it more dangerous in engineering. It does not announce itself, so it can pass review when the surrounding work looks correct. The defense is the same either way: verify values against primary sources and keep a qualified human accountable for the result.

Sources

• Kalai et al., Why Language Models Hallucinate (arXiv 2509.04664, 2025)
• Mirzadeh et al., GSM-Symbolic (arXiv 2410.05229, 2024)
• Walters and Wilder, Fabrication and errors in ChatGPT citations (Scientific Reports, 2023)
• Dahl et al., Large Legal Fictions (Journal of Legal Analysis / Stanford, 2024)

Written by the CognitiveFuture editorial team. We build our guidance from published research and official documentation, and we attribute every figure to its source and its model generation. We do not independently benchmark any tool, and we do not treat AI-generated numbers, citations, or standard clauses as reliable. Any output that feeds a stamped design, a code-compliance decision, or a purchase must be verified against the primary source and reviewed by a licensed engineer.