A recruiter asks an assistant for a short background sketch on a candidate ahead of tomorrow's interview. Back come four fluent paragraphs: a degree, two previous employers, a date. Three of those four facts are right. The fourth is invented, and it is stated with precisely the same confidence as the other three.
Understanding why means understanding what happens when a model answers. It is not retrieval.
The model predicts, it does not consult
A language model holds no fact database it consults. During training it learned patterns from enormous quantities of text, and those patterns exist as probabilities: given these tokens, what is the most likely continuation?
Every answer is assembled that way. Token by token, always the same question: what fits best here.
Where the patterns are strong, because something appeared thousands of times the same way in training data, a correct answer usually falls out. Where the patterns are weak or contradictory, out comes something that matches the shape of an answer without matching reality.
That is the crux: both answers come from the same process. There is no separate mechanism for "I am certain" and "I am guessing". The model cannot see the difference internally, so it cannot report it to you.
Why it sounds so sure
Assertiveness is a pattern too. Encyclopedias, textbooks and news reports use declarative sentences, and that is what the model learned from. So it writes declaratively, whether or not the content holds up.
There is a second effect. Models are tuned after training on human preference, and people generally rate a helpful, fluent answer above "I don't know". That preference is baked into the result. Admitting a gap is trained behaviour competing against a much stronger pull toward being useful.
So confidence carries no information about reliability. That is genuinely hard to internalise, because with people it usually does carry information.
Where it goes wrong most often
In practice the inventions cluster in predictable places:
- Names, dates and numbers. Precise details about specific people are rarely well represented in training data.
- Citations. A real author, a plausible title, a paper that does not exist. The shape of a citation is an easy pattern to reproduce.
- Legislation. Article numbers and recitals are trivially imitable without being right.
- Recent events. Anything past the training cutoff, unless the tool actually searches.
- Anything combinatorial. Two accurate facts joined into a relationship that was never there.
Tools that search first and summarise second do better here, because there is a source to point at. Even then: open the link. A citation that looks right is not the same as a citation that supports the claim.
Then the data protection side
This is where it stops being a quality problem.
When a model asserts something untrue about an identifiable person, that is processing of personal data that is inaccurate. Article 5(1)(d) GDPR requires personal data to be accurate and inaccurate data to be erased or rectified without delay. Article 16 gives the data subject the right to demand it.
Not theoretical. The privacy group noyb has brought complaints against OpenAI on this basis, including a case where ChatGPT stated that a Norwegian man had murdered his children. An earlier complaint concerned a fabricated date of birth, where the sharper point was that the provider could not correct the specific inaccuracy.
That last part is the structural difficulty. A fact in a database is corrected by editing the row. A pattern inside a model is not: you can suppress the output, you can tune around it, but removing one incorrect fact is a different operation from rectification as the GDPR imagines it.
What this means at work
Two things, and the second one gets forgotten.
Check what you carry over. Every figure, date, name and legal reference. Not because the model is unreliable across the board, but because nothing in the answer tells you which part needs checking.
Watch what you record. This is the side rarely discussed. When an AI answer about an applicant, an employee or a customer lands unchecked in a file, the invention is no longer a third party's output. It is your recorded personal data, in your system, under your controllership. The rectification duty moves to you at that moment.
Once that lands, people use AI differently for text than for people. Rewriting a paragraph only needs a fluent answer. Anything about a person that gets stored needs a different question: not whether it reads well, but whether someone checked it before it stayed.
