A recruiter asks an assistant for a short background sketch on a candidate ahead of tomorrow's interview. Back come four fluent paragraphs: a degree, two previous employers, a date. Three of those four facts are right. The fourth is invented, and it is stated with precisely the same confidence as the other three.
To understand why, you need to know what happens when a model answers. It is not looking anything up.
The model predicts, it does not consult
A language model has no database of facts to consult. During training it learned patterns from enormous quantities of text, and those patterns exist as probabilities: given these tokens, what is the most likely continuation?
Every answer is assembled that way. Token by token, always the same question: what fits best here?
Where the patterns are strong, because something appeared thousands of times in the same way in the training data, a correct answer usually falls out. Where the patterns are weak or contradictory, out comes something that matches the shape of an answer without matching reality.
That is the crux: both answers come from the same process. There is no separate mechanism for "I am certain" and "I am guessing". The model cannot see the difference internally, so it cannot report it to you.
Why it sounds so sure
Assertiveness is a pattern too. Encyclopedias, textbooks and news reports use declarative sentences, and that is what the model learned from. So it writes declaratively, whether or not the content holds up.
There is a second effect. After training, models are tuned to human preferences, and people generally rate a helpful, fluent answer above "I don't know". That preference is baked into the result. Admitting a gap is trained behaviour competing against a much stronger pull towards being useful.
So confidence carries no information about reliability. That is genuinely hard to take on board, because with people it usually does.
Where it goes wrong most often
In practice the inventions cluster in predictable places:
- Names, dates and numbers. Precise details about specific people are rarely well represented in training data.
- Citations. A real author, a plausible title, a paper that does not exist. The shape of a citation is an easy pattern to reproduce.
- Legislation. Article numbers and recitals are easy to imitate without being correct.
- Recent events. Anything past the training cutoff, unless the tool actually searches.
- Anything combinatorial. Two accurate facts joined into a relationship that was never there.
Tools that search first and summarise second do better here, because there is a source to point at. Even then: open the link. A citation that looks right is not the same as a citation that supports the claim.
Then there is data protection
This is where it stops being just a quality problem.
When a model asserts something untrue about an identifiable person, that is the processing of inaccurate personal data. Article 5(1)(d) GDPR requires personal data to be accurate and inaccurate data to be erased or rectified without delay. Article 16 gives the data subject the right to demand that.
This is not theoretical. The privacy group noyb has brought complaints against OpenAI on this basis, including a case where ChatGPT stated that a Norwegian man had murdered his children. An earlier complaint concerned a fabricated date of birth, where the sharper point was that the provider could not correct the specific inaccuracy.
That last part is the structural difficulty. A fact in a database is corrected by editing the row. A pattern inside a model is not: you can suppress the output, you can tune around it, but removing one incorrect fact is a different operation from rectification as the GDPR envisages it.
What this means at work
Two things, and the second is the one people forget.
Check what you reuse. Every figure, date, name and legal reference. Not because the model is unreliable across the board, but because nothing in the answer tells you which part needs checking.
Watch what you record. This is the side that rarely gets discussed. When an AI answer about an applicant, an employee or a customer lands unchecked in a file, the invention is no longer a third party's output. It is personal data you have recorded, in your system, with you as controller. From that moment, the duty to rectify it is yours.
Once that sinks in, people start to treat AI help with wording differently from AI answers about people. Rewriting a paragraph only needs a fluent answer. Anything about a person that gets stored needs a different question: not whether it reads well, but whether someone checked it before it was kept.
