Ask a language model a question it cannot answer and it will usually answer anyway. Fluently. The text looks like every other answer it gives. This is the single property of these models that matters most for business use, and the one least talked about in a sales pitch.
It is not lying, because there is no intent. The closest word is confabulation: filling a gap with something plausible, the way a person does when they are sure they parked on level two.
Why it happens
The model is built to continue text well. "I do not know" is a poor continuation of most questions, so it is rarely the one chosen unless the model has been shown, many times, that stopping is the right move here. Left to itself, it will prefer a confident wrong answer to an honest blank.
What a system does about it
Three things, none of them a bigger model.
Give it somewhere to look. When the answer should come from the business's own files, the system looks there first and answers from what it finds. If it finds nothing, that is the answer: nothing was found, so a person will follow up.
Give it permission to stop. The page of limits says what it does not answer: prices, advice, anything about a claim or a case. The model is not asked to judge whether it knows. It is told, in advance, where it does not.
Check it afterwards. A second pass reads what the model wrote and checks it against the source. A document's fields are checked against the document. A call's record is checked against the transcript. A claim that cannot be traced is dropped before it reaches anyone.
What this feels like to a customer
Slightly less magical and much more trustworthy. The assistant says "I will have someone call you about that" a little more often than a demo would. In exchange, nobody gets told a price that does not exist.
There is a longer piece on this on the blog: confabulation, not hallucination.