AI hallucinations — what they really are, why they sound so confident, and how to stay in control

When an AI model states something false, it usually does so in the same confident tone as the truth. It doesn’t hesitate, doesn’t signal uncertainty — it simply assembles a sentence that sounds plausible. That is why AI hallucinations are dangerous not in themselves, but through how easy they are to fall for. This text explains what a hallucination really is, why you can’t “switch it off,” and what to actually do so you don’t get caught by it.

What it actually is — no mysticism

A hallucination is not a malfunction, not the model being “broken.” It is a property of how the model works. A language model does not check facts in a database — it assembles the most probable answer from the patterns it learned. When it knows, it assembles a fact. When it doesn’t know, it assembles something that looks like a fact — with the same confidence. From its perspective, it’s the same operation. That is why it can’t warn you “I’m making this up here” — because on its side there is no difference between the two.

This matters, because it changes how you defend against it: if a hallucination is a property, not a defect, then you can’t patch it away with an update. You can, however, stay in control of it — more on that below.

Two different ways AI departs from the truth

It’s worth separating two things the market often lumps together as “AI lies,” because they are treated differently:

1. A true hallucination. The model doesn’t know and assembles a fabrication instead of admitting ignorance. The remedy: verification and a human checking the result.

2. A role artifact. This is subtler and less well known. When you tell a model “act as an expert,” “be a lawyer,” “answer like a specialist” — it starts playing that character. And good character-play means: speak confidently, don’t admit ignorance, fill in in character — including untrue things, because “an expert would know.” Assigning a role can therefore raise both the model’s confidence and its error rate at once. The remedy is different from the one for hallucination: instead of giving a role, give a concrete task with a clear start and end. A role tells it to pretend; a task tells it to deliver.

This distinction has practical weight: some of the “hallucinations” companies complain about are in fact artifacts of a badly set role — and they disappear once you stop telling the model to be someone and start telling it to do something.

The scale — hard data, not anecdotes

That this is not a marginal problem is shown by numbers from serious sources:

• Stanford (RegLab / HAI) tested general-purpose models (GPT-3.5, Llama 2, PaLM 2) on legal queries in January 2024: they hallucinated in 69–88% of cases. In a May 2024 study, even dedicated “legal” AI tools were wrong 17–33% of the time — that is one in six to one in three questions, in a field where an error is costly.
• Hallucinations already have a footprint in courts: over 1,170 cases of citing AI-fabricated content in proceedings have been documented (including hundreds in the US since 2025), with over 120 invented legal citations. Sanctions for it escalated roughly elevenfold in a year and a half.

You can’t switch it off — you can stay in control

Since a hallucination is built into how the model works, the goal is not “AI that never errs” (a promise no honest person will make), but a layer of control over the error. In practice:

• Verification by someone other than the author. Treat the model’s output as a proposal to check, not an oracle. Most effective when an independent person or an independent system checks it — not the same one that generated it.
• A human in the loop where the stakes are high. The more serious the consequence, the less room for “it probably works.”
• Task boundaries. A model kept in a narrow, concrete scope hallucinates less than one asked for “everything about everything.”
• Demand a source. “Tell me how you know this” — and check whether the source exists — catches a good share of fabrications, especially those delivered in a confident tone.

Summary

A hallucination is not a bug but the price of how a model assembles answers — which is why it sounds so confident and why you can’t simply switch it off. Hard data (Stanford: 69–88% on legal queries; over 1,170 court cases) shows this is a real, costly problem, not a curiosity. But it is manageable: distinguish a hallucination from a badly-set-role artifact, keep the model within task boundaries, and verify the output instead of trusting it.

A mature approach to AI is not about promising a “model without errors.” It is about honestly saying: an error will happen — and building the layer that catches it before it becomes your cost.

---

MafiaAI — a team of people and AI agents building tools, websites and solutions. We don’t promise AI without errors — we build control over the error: verification and oversight built into the solution. More: t8.pl

Sources: Stanford RegLab / HAI, “Hallucinating Law: Legal Mistakes with Large Language Models Are Pervasive” (2024) — hallucinations from 69% (GPT-3.5) to 88% (Llama 2) on specific legal queries (January 2024 study, general-purpose models); Stanford, “AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries” (May 2024) — dedicated legal tools: about 17% (Lexis+ AI, Ask Practical Law) to about 33% (Westlaw); public registries of court cases involving AI-fabricated content.