Why Language Models Invent Sources, and What Actually Stops It

By Pooja Goenka · Published 2026-09-29 · Updated 2026-10-21

In 2023 a lawyer in New York needed case law for a filing and asked ChatGPT to find it. It came back with six decisions. He was suspicious enough to check, so he did the thing that feels like checking: he asked the chatbot whether the cases were real. It said yes. It added that he could find them on Westlaw and LexisNexis.

He filed them. Opposing counsel could not locate the cases. Neither could the judge. None of the six existed. The court sanctioned the lawyer, a colleague and their firm five thousand dollars, and the opinion described the reasoning inside one of the invented decisions as gibberish. The case is Mata v. Avianca, decided in the Southern District of New York on 22 June 2023, and you can read the opinion yourself.

The story usually gets told as a warning about trusting AI. I think that reading misses the useful part. Nothing malfunctioned that day. The model did what it is built to do, and a citation is close to the worst thing you can ask it for. I want to explain why properly, because the same mechanism sits underneath anything we build on these models.

The explanation that sounds right and is not

The common account is that the model lied, or hallucinated, or was overconfident. Those words all smuggle in a mind that knew something and chose to misreport it.

A language model has nothing like that inside it. It does not hold a set of facts and consult them. It holds a very large statistical picture of which token tends to follow which other tokens, and at every step it produces a probability for every token in its vocabulary and picks from that distribution. That is the whole mechanism. Once you see it that way, the invented citations look less like a fault and more like the expected result of picking likely tokens.

Sharp distributions and flat ones

Give a model an unambiguous context and the distribution has one obvious winner. For "The capital of France is ___", Paris takes about 94 percent and the rest of the vocabulary splits a few points between them. The model looks like it knows a fact. What it has is a very lopsided distribution, which for practical purposes is the same thing.

Now give it an open context. "After the meeting, Anjali was feeling ___" produces nothing like a winner. Tired might be 12 percent, frustrated 10, optimistic 9, overwhelmed 8, and nearly half the mass scattered across thousands of other plausible words. There is no fact to retrieve here, and the model is not in any difficulty. It samples something reasonable and moves on, exactly as designed.

The trouble is that from the outside those two situations produce identical looking output, a fluent sentence in the same steady tone, and nothing in the result shows you which kind of distribution it came from.

A case citation is a flat distribution that looks like a sharp one. "Varghese v. China Southern Airlines Co., 925 F.3d 1339 (11th Cir. 2019)" follows a tightly constrained format: two party names, a volume number, a reporter, a page, a circuit and a year. The model has seen tens of thousands of citations and has a precise sense of what one looks like. What it does not have, unless someone put it there, is the actual reporter. So it fills a well understood shape with plausible contents, and the result is a string that passes every test a reader applies by eye.

Why asking it to double check cannot work

The lawyer did try to verify. He asked the model directly whether the case was real, and it confirmed it and named the databases where he would find it. Most retellings skip that detail, and I find it the most useful part of the story.

That confirmation was generated the same way the citation was. "Is Varghese a real case?" is a context, and given everything that came before it in the conversation, "Yes, it is a real case" is an extremely likely continuation. The model was not consulting anything. It had no more access to Westlaw in that second message than it had in the first. Asking a model whether its own output is true is asking it to predict what a confirmation would sound like, and it is very good at that.

So he did check, but with a method that could never have caught the problem. Nothing in that conversation could look the case up. He was asking the same generator a second question and getting another likely continuation back.

What retrieval actually changes

The fix is unglamorous and mostly happens outside the model. Before the model answers, you search a real source you control, pull back the specific documents that bear on the question, and put them into the prompt. Then you ask the model to answer from those. That is retrieval augmented generation, and most of the work in it is the retrieval step.

The model is the same next-token predictor before and after, no more truthful than it was. The context it predicts from is what changes. When the relevant passage is sitting in the prompt, the continuation that reproduces that passage is now the high probability one, and the invented alternative is competing against a real document instead of against nothing.

That also explains the failures you see afterwards. A grounded system still gets things wrong, in more tractable ways: retrieval returns the wrong passage, or returns nothing and the model answers anyway, or returns the right passage and the model summarises it badly. You can name each of those, write a test for it and fix it. You cannot do any of that for "the model invented a source", because no single step went wrong.

Where the model's own memory is still the right tool

Plenty of tasks do not need a retrieval pipeline at all.

For anything the model absorbed in enormous volume and that has not changed since, parametric memory is fine and much cheaper than a search: language itself, common algorithms, and the ordinary work of summarising, rewriting, translating and classifying text you have already handed it. In all of those the model is either working on what is in front of it or on patterns so widely attested that the distribution really is sharp.

My own rule is that if being wrong about a specific detail matters, and that detail lives in a source you could look up, look it up. That covers identifiers, prices, policies, dates, names, anything from your own systems, and anything from after the training cutoff. If you could not check the claim yourself without opening a document, the model could not either.

What to take from it

Assume a specific, checkable claim from a bare model is unverified until you have checked it against the source yourself. Asking the model again does not count.

Notice when you are requesting a fact and when you are requesting prose. The two feel like the same request, and the output will not tell you which one you made.

If you are building something where the facts come from your own data, put retrieval in from the start. Retrofitting it after someone notices the invented answer is the expensive order to do it in.

To build the fix yourself, the RAG tutorial in Python runs offline with no API key, and RAG vs fine-tuning covers when retrieval is the wrong tool.

Where you can watch this happen

We built the LogicWiz Generative AI course around this problem, because anyone shipping a product on these models runs into it sooner or later.

One lesson puts the probability distribution on screen so you can switch prompts and watch it go from a single 94 percent spike to a flat spread across a dozen candidates. Another runs the same question through the same model twice, once with nothing and once with a single retrieved support ticket, and you read the two answers side by side. The retrieval chapter goes through it properly: why Nova needs RAG, embeddings, chunking, finding the right passage, and what to do when the retrieved passage is the wrong one.

The course is completely free. The early chapters open with no account at all, and a free account keeps you going through the later ones. There is no card at any point.

The court opinion is worth twenty minutes of your time too. It is a clear account of how plausible text from a model ended up in a court filing as if it were a source.