Why do AI talk nonsense with a serious look?

How Strong Is AI's "Fabrication" Ability?

In 2016, there was a famous news story: Microsoft's chatbot Tay was "corrupted" by netizens less than 24 hours after going live, and began frantically posting racist remarks. Microsoft quickly took it offline.

But Tay's problem was "obvious"—it was maliciously steered. Another problem is more insidious: AI sometimes, in a very confident and fluent tone, tells you something completely wrong.

Ask it, "What is the speed of light?" and it may give you a number precise to ten decimal places, which looks very professional—but what if that number is wrong?

This is AI hallucination, the "talking nonsense with a straight face" of the AI world.

What Is AI Hallucination?

In simple terms, AI hallucination is: AI-generated content that looks reasonable and fluent, but is actually wrong, fabricated, or unfounded.

It has three characteristics:

1. Confidence: AI won't say "I'm not sure"; it will tell you with absolute certainty.
2. Fluency: Grammatically correct and logically consistent, so it seems flawless at first glance.
3. Error: Content that doesn't match the facts—possibly numerical errors, citation errors, or logical errors.

Why Do Hallucinations Occur?

The reasons for AI hallucinations can start with its "genes."

Reason 1: The Essence of Language Models Is "Prediction"

Remember what we said? The core task of large language models is to predict the next word.

It isn't "recalling" the correct answer; instead, it "stitches together" the most plausible sentence based on language patterns.

Like a student who has memorized many model essays, when encountering an unfamiliar exam question, the student isn't thinking "What is the correct answer?" but rather "How are these kinds of questions usually answered?"—sometimes turning out something plausible.

Reason 2: "Noise" in Training Data

AI learns from massive internet data, and the information on the internet is not always correct.

AI may pick up a wrong viewpoint from a forum post, then cite it in its next answer—and cite it "quite plausibly."

Reason 3: Lack of "Truth Perception"

Humans know what they "know for sure" and what they are "not sure about."

But AI lacks this perception. It cannot distinguish between "I actually learned this fact" and "I just saw similar expressions in the corpus."

So when asked a question with fuzzy boundaries, it will "do its best to make one up"—rather than saying "I don't know."

Reason 4: Contextual "Overreach"

Sometimes the question itself hints at a certain answer. To "cooperate" with your question, AI may follow your hint and fabricate.

You ask: "When was Tesla CEO Musk born, and from which university did he graduate?" (Actually, Musk never dropped out of Stanford, but this question presupposes that he did.) AI may not correct this false premise, but instead fabricate the entire answer along those lines.

What Types of AI Hallucinations Are There?

1. Factual Hallucinations

Generated content that does not align with objective facts.

• Example: Saying "the Earth is flat" (incorrect fact)

2. Fabricated Hallucinations

Generated content that is completely invented, such as non-existent book titles, non-existent papers, or non-existent dates.

• Example: Citing a paper that doesn't exist at all to support an argument.

3. Logical Hallucinations

Content that is self-contradictory and violates basic logical rules.

• Example: First saying "seawater is salty," then saying "seawater is drinkable because it's fresh."

4. Domain-Specific Hallucinations

Providing answers that look professional but are completely wrong in a specialized field.

• Example: Giving an incorrect drug formula.

How to Deal with AI Hallucinations?

As a User, You Can:

1. Verify important information: especially in sensitive areas such as health, law, and finance.
2. Ask for sources: Have AI state its information source—"How do you know that?"
3. Verify in segments: Don't ask AI to write a long article all at once; confirm section by section.
4. Cross-verify: Use multiple AIs or multiple channels to check information.
5. Allow "I don't know": When asking AI uncertain questions, require it to "say so if unsure."

As a Developer, You Can:

1. Integrate RAG: Let AI answer "while consulting materials" to reduce fabrication.
2. Prompt engineering: Clearly tell AI "don't fabricate" and "say 'I don't know' if you don't know."
3. Fine-tuning: Conduct targeted training for scenarios prone to hallucination.
4. Confidence labeling: Have AI give a "reliability level" for each part of its answer.
5. Post-processing verification: After AI answers, use another model to check for errors.

Is Hallucination a "Bug" or a "Feature"?

Interestingly, sometimes AI's ability to "fabricate" is actually valuable:

• Creative writing: Have it invent a non-existent world view or historical legend.
• Brainstorming: Generate all sorts of possibilities, including absurd ones.
• Language practice: Having students find errors that AI made is actually a learning opportunity.

So hallucination is not necessarily a bad thing; it depends on the application scenario.

What Does the Future Hold?

AI researchers are working hard to solve the hallucination problem:

• Retrieval-Augmented Generation (RAG): Let AI answer "while looking at materials" to reduce fabrication.
• Chain-of-Thought: Let AI show its reasoning process to make errors easier to spot.
• Tool use: Let AI proactively search or query databases when uncertain.
• Multi-AI verification: Use multiple AIs to check each other and catch each other's mistakes.

But fundamentally, hallucination may not be completely eliminated—because it is rooted in the nature of language models. What we can do is make it more controllable and more transparent.

Summary

AI hallucination is an "innate flaw" of language models—its essence is "predicting the next word" rather than "recalling the correct answer," plus the noise in training data and lack of truth perception, causing it to often talk nonsense with a straight face. Dealing with hallucination requires joint efforts from users and developers—users should stay alert and verify information, while developers can reduce hallucination risks through methods like RAG and prompt engineering.

Next time, let's talk about a more serious topic—AI ethics and safety, and look at the risks and challenges facing AI development.