A language model generates the most likely text, not the most truthful one. It can therefore cite a function that doesn’t exist, invent a source or state a false fact with confidence.
For a QA team, testing an AI application starts with measuring this hallucination rate on questions whose answer is known, then checking that the application knows how to say “I don’t know”.