AI hallucinations undermine human judgment and confidence

Recent research has found that AI hallucinations don’t just deceive you—they can also inflate your confidence and undermine your judgment.

That's not good.

Recently, a study by École Normale Supérieure de Paris, Sapienza University of Rome, and the University of Milano-Bicocca found:

Without AI, people are more willing to honestly admit, "I don't know," and hold off on making a judgment;

But as soon as AI appears, most people will accept its answers as their own judgments and confidently voice them, even though they know AI can make mistakes and that incorrect answers may lead to penalties.

The experimental results also clearly show this trend: in the first experiment without AI, 44% of participants admitted "I don't know," with a correct answer rate of 27%;

After using AI assistance, the percentage of people who said they didn't know dropped sharply from 44% to 3%, accuracy fell from 27% to 9%, but confidence surged from 30% to 76%.

Even though subsequent experiments attempted to encourage participants to maintain independent judgment and correct this tendency through a reward-penalty system—rewarding correct answers and penalizing incorrect ones—people still struggled to overcome their reliance on AI’s erroneous advice.

In other words, those who don’t understand haven’t truly disappeared; instead, they’ve been replaced by a group emboldened by AI to confidently give incorrect answers.

The related research findings have been published on arXiv. The team conducted five experiments with a total of 3,132 participants, four of which were pre-registered, and one was a direct replication.

This study examines the impact of AI on human judgment.

Researcher Valerio Capraro said in an interview:

For humans, saying "I don't know" is crucial because it shows we recognize the limits of our knowledge.

But now, with AI, almost any question can immediately receive a fluent answer.

So the question arises: could this interfere with humans’ natural ability to judge—to hold off on drawing conclusions when uncertain?

To this end, the research team designed a set of questions that large language models are likely to answer incorrectly.

It's not a math reasoning question, nor a common sense question, but a visual detail question from a movie.

For example, what color is the team jersey in "Bend It Like Beckham"? What is Agatha's signature hairstyle in "The Grand Budapest Hotel"? Or what car does Monica drive in "A Cat on the Highway"?

Do you remember? I was completely stuck for ages and couldn’t recall it—this is way too obscure! Who could possibly remember this... (shrugs)

For AI, this would also be extremely challenging, as the answers to these questions are often not found in conventional text materials, and the model’s training data likely lacks these visual details.

Researchers first tested the model used in the experiment, Step 3.5 Flash, and found the results as expected—it typically answers incorrectly.

In addition, they tested more advanced models, including GPT-5.5, Claude Sonnet 4.6, and Gemini 3.5 Flash.

These advanced models do perform better on certain questions, such as Agatha’s iconic hairstyle in The Grand Budapest Hotel—GPT-5.5, Gemini 3.5 Flash, and Claude Sonnet 4.6 all point in the correct direction: crown braid, milkmaid braid—meaning a braid that wraps around the head.

△

△

△

For example, in "Bend It Like Beckham," the team uniform colors also largely capture key information such as white and red.

But when it comes to questions that rely more on specific visual memory and are less covered by online textual data, the models begin to fail collectively—and each failure is uniquely impressive.

The most typical example is the car driven by Monica in "A Cat on the Highway." The correct answer is a 2008 blue Toyota Aygo.

But GPT-5.5 described it as a scooter and added a narrative that “she rode it around the suburbs of Rome and later used it to deliver pizzas after opening her own pizzeria,” along with details such as “the contrast between her modest, working-class lifestyle and Giovanni’s affluent urban life.”

It looks logical and reasonable, but it’s completely made up by the model just to make you think it’s plausible! (Got fooled, huh hhh)

Gemini 3.5 Flash said she primarily takes the bus and doesn’t own a car, with even more numerous and subtle errors (for example, there is no consistent pattern in the narrative that Monica regularly rides with Sergio—the model incorrectly generalized scattered instances of hitching rides into a nonexistent habit):

△

Claude Sonnet 4.6 inferred the 2016 Toyota Auris Touring Sports Hybrid from the online automotive database:

△

Step 3.5 Flash dropped the pretense and directly claimed it was a 2007 Fiat 500 (come on, that’s obviously fake!).

The problem with these answers isn't just that they're wrong—they're wrong in a way that feels convincingly true, making them hard to distinguish!

They are not merely ignorant—they fabricate character backgrounds, invent plot logic, and invent vehicle purposes to present their incorrect answers in a complete, seemingly logical manner, yet the core facts are wrong.

This is precisely where AI hallucinations are most dangerous—when the core facts are the most critical point, it masks its ignorance and errors with fluent phrasing, making you believe it understands everything.

The experiment was primarily conducted using the lighter Step 3.5 Flash, rather than selecting top models like Claude or GPT.

Design six questions about movie details, which is a crucial aspect of research design, as many people know that AI frequently answers such questions incorrectly or unreliably.

Researchers said they did this to rule out one explanation: that "people follow AI because AI is genuinely reliable."

If people knowingly follow AI’s answers despite being aware that AI frequently gets them wrong, the issue is no longer just “reasonably outsourcing judgment”—it’s that their ability to judge has truly been suppressed by AI.

The research is divided into five rounds.

In the first two rounds of experiments, we primarily examined whether AI recommendations affect the proportion of "I don't know" responses.

In Study 1a, 314 participants were divided into two groups to answer six movie detail questions. (One group had no AI access, while the other could choose whether to seek help from AI.)

The results are clear: without AI, 36% of participants chose to withhold judgment; with AI available, this percentage dropped to 6%.

In other words, as long as they can ask an AI, people clearly prefer not to admit they don't know.

However, researchers are concerned about a technical issue, as in Study 1a, approximately 10% of the AI tool's requests did not respond properly.

So they conducted Study 1b.

This time, researchers did not call on AI in real time; instead, they pre-generated AI responses to ensure that each requester could see the complete answer.

The result was replicated: without AI, 44% of people said "I don't know"; with AI, that number dropped to just 3%.

Capraro called this phenomenon "the collapse of verification suspension." (Tricked by AI's serious demeanor, one forgets to check whether its underlying assumptions and various premises are correct.)

Next, the researchers asked: Would people be more cautious if there were consequences for getting the answer wrong?

Study 2 introduced a monetary reward and penalty system.

This round involved 812 participants, divided into four groups based on the presence or absence of AI recommendations and monetary incentives or penalties.

The rule is: correct answers earn $0.10 (approximately RMB 0.73), incorrect answers deduct $0.10, and saying "I don't know" results in no reward or deduction.

Logically, although the amount of money isn’t large, there’s still a cost to getting it wrong, so people should be more careful.

However, the results showed that money did provide some help, but it was far from enough to offset the impact of AI:

Without monetary incentives or penalties, the use of AI can reduce the percentage of "I don't know" responses from 17% to 1%.

Even with monetary incentives and penalties in place, AI availability still reduces this percentage from 21% to 2%.

In other words: The experimental results show that, even when people know they will be penalized for incorrect answers and are aware that AI is not very reliable on such questions, most still choose to trust the AI’s answer over their own judgment.

More importantly, accuracy.

Without AI, the accuracy rate was approximately 28%; with AI, the accuracy rate dropped to just 10%.

This shows that some people, despite being aware that AI can hallucinate, still accept the answers provided by AI.

After introducing monetary incentives and penalties, accuracy improved:

Without AI: from 28% to 33%; with AI: from 10% to 17%.

Although the accuracy rate of the AI group is still significantly lower than that of the non-AI group, this rule is not entirely ineffective—it at least reduces the number of times people turn to AI for help:

Data shows that, under incentive conditions, the average number of times participants sought advice from AI decreased from 5.44 to 4.93.

In other words, money may slightly reduce people’s reliance on or trust in AI, but it cannot fully free them from the influence of AI’s erroneous recommendations, and it is difficult to bring people back to a state of cautious judgment.

Most thought-provoking is the inflated confidence people exhibit when AI gives incorrect answers:

Without AI and without reward or penalty rules, participants' average confidence was 29.6%.

After introducing AI, participants' confidence surged directly to 75.9%; additionally, even with monetary incentives and penalties in place, the AI group's confidence remained at 73.5%.

The paper's summary is straightforward: AI makes people more confident, but not necessarily more accurate.

AI is integrating into daily life at an unprecedented pace, and one obvious change is that many AI suggestions now come to us not because we asked for them, but because the AI proactively offers them.

The most typical scenario is trending on social media.

Previously, to dive into a topic, you had to manually navigate through comments, media reports, and statements from those involved, piecing together the timeline as you went—while occasionally stumbling upon unrelated gossip along the way.

It’s different now—when you first enter, the AI-generated summary is already prominently displayed at the top.

I’ve laid out everything for you: what happened, who said what, the points of contention, and how things have progressed—complete with links to the original news sources.

It's certainly convenient, but also a bit subtle...

The clearer it organizes the information, the shorter our own process of exploring the background becomes, and the joy of previously rummaging through everything, making our own judgments, and stumbling upon unexpected details seems to have been compressed away as well.

When writing, before you've even typed two lines, AI-generated completions or suggestions are already following the cursor.

Therefore, Study 4 adopted a more realistic design: instead of letting participants choose whether to ask the AI, the AI’s suggestion was automatically displayed next to each question.

This round had 853 participants, and the results were largely consistent with the previous ones.

Without monetary incentives or penalties, AI's automatic appearance reduces the "I don't know" rate from 35% to 1%; with rules in place, it decreases from 39% to 7%.

Accuracy continues to be affected: without AI, accuracy was approximately 27%; with AI but no incentives, accuracy dropped to only 7%.

With AI and incentives added, the accuracy rate improved to 14%.

This shows that even without actively seeking AI assistance, merely being exposed to AI recommendations can strongly influence people.

This is a signal that must be watched closely—AI is quietly reshaping our judgment habits.

It’s no longer just a tool that responds only when you initiate a chat—it has become the default “standard answer” pushed to you by the system.

• It may appear at the top of the search results page.
• Appears in the email draft.
• Appears in the document sidebar.
• Appears in learning software, office systems, and browser extensions.

Once the answer is automatically presented before us, the human judgment process has already been rewritten; but the most terrifying part is that we’ve grown accustomed to it.

The paper used the term: epistemia (cognitive inertia)

People may accept the surface credibility of AI-generated answers because they appear fluent, complete, and well-structured, rather than verifying them further.

A large language model has a fundamental characteristic: it always generates something. Even when faced with questions it doesn't know, it rarely truly stops.

It can sound very convincing. But if we truly choose to delegate judgment to such a system, we may inherit its habit of “not pausing.”

One explanation provided in the paper is that "the fluent answers generated by AI lower the threshold at which humans decide, 'I know enough to answer.'"

You might originally think: "I don't know this question, I'm not sure—never mind... I need to figure it out before I can answer."

But after AI gives you a statement that seems very certain, the psychological barrier shifts to “Since it said that, I can answer too.”

Thus, "I don't know" and the process of inquiry were omitted and replaced with a "standard but incorrect answer," and this incorrect answer granted you a high level of confidence that should only come from repeated exploration and verification.

Even if answering incorrectly incurs a fine, people only become slightly more cautious and cannot completely overcome their habit of overly trusting AI, resulting in being thoroughly misled.

What this study truly reminds us of is not that “AI hallucinates”—something that is already well known—but rather a concerning new development: AI is changing how humans handle uncertainty.

In the past, when faced with an unfamiliar problem, a person might have had three options:

If you know, answer.

2. If you don't know, look it up.

3. If you're not sure, don't answer or make a judgment.

But after adding AI, the process became shorter.

1. AI provides the answer.

2. The user accepts the answer.

The most crucial step, "Do I actually know?", has been inadvertently skipped.

This is why the paper's authors emphasize that their study is not about "what answers AI prompts people to give," but rather "how AI makes it easier for people to decide 'I can answer.'"

This goes a step deeper than simply getting one answer wrong, because it erodes our metacognitive ability—the internal alarm system that monitors the boundaries of our knowledge, which is incredibly valuable.

When this alarm fails, we lose our instinct to distinguish between "true knowledge" and "plausible guesses."

At that time, we may still be able to efficiently obtain answers through AI, but we will no longer be able to determine whether those answers truly belong to us or are genuinely trustworthy.

Meanwhile, what AI leads us astray from will no longer be a single specific judgment, but the very foundation of our ability to think independently.

This loss is incalculable.

One of the paper’s authors, Valerio Capraro, said in an interview: “I am very concerned about children, because adults have already learned critical thinking. But for children who have grown up with these systems from birth, the risk is that they may never learn basic critical skills.”

Behind this statement lies a deeper concern: today’s adults have at least lived in a world without AI.

So, by first learning basic judgment skills through search engines, books, classrooms, debates, and trial and error before using AI, one has the opportunity to develop a sense of “I’m not sure.”

But today’s children may grow up alongside these AIs from birth.

They might casually ask AI for help with homework, researching information, writing articles, or understanding concepts.

If one gets into the habit of simply receiving answers from the start, rather than forming their own judgment first and then verifying the answer, the risk is not just getting one question wrong—they may never develop the most basic critical thinking skills.

In Capraro’s view, solving this issue cannot rely solely on model companies fixing bugs; it must also address AI literacy and education policy.

Model companies should certainly improve, for example, by better expressing uncertainty, reducing hallucinations, and reminding users when evidence is insufficient.

At the same time, Capraro also believes that one cannot place all hopes on model companies.

A more promising and long-term solution is education.

Especially children's education.

For today’s children, practicing and cultivating the habit of saying, “I don’t know, but I’ll find out,” is essential and important.

At the end of the experiment, the researchers also acknowledged limitations: the study used only movie trivia questions, so it remains to be verified whether the findings apply to other domains; the incentive amount was small, and it is unclear whether greater stakes could further correct behavior.

But at least the existing data has sent a clear signal: people are saying "I don't know" less often, giving answers more frequently, getting fewer correct, yet becoming more confident.

This could be one of the most hidden risks of the AI era.

AI not only deceived you, but also made it harder for you to realize that you actually don't know.

The study’s final judgment is restrained but sobering: “As AI-generated answers become ubiquitous—and often unsolicited—our research suggests that humanity’s ability to comfortably admit ‘I don’t know’ may be among the earliest casualties in human-AI interaction.”

As AI systems become increasingly widespread, whether human judgment can hold its ground may ultimately depend not on making AI more accurate, but on our ability to maintain self-awareness—clearly recognizing the limits of our own knowledge and acting with caution accordingly.

This doesn't mean AI can't be used; on the contrary, AI can certainly be a powerful auxiliary tool;

But on the condition that it helps people think, rather than replacing their thinking, we must uphold our底线 of cautious judgment.

A healthier way to use AI is to first judge for yourself, take the initiative to learn, and only seek help from AI when you truly can’t figure it out—while maintaining a critical mindset throughout the process (questioning both the AI and yourself).

For example:

Could it be wrong?

Where is the evidence?

Is the reference link returning a 404 or blank page?

Did I assume it was right just because it sounded so smooth?

This is also a wake-up call for AI products and their developers: "The maturity of technology will ultimately be reflected in reverence for the unknown."

The future direction of AI evolution may not only be to make AI respond faster and more human-like, but also to teach AI to honestly acknowledge uncertainty;

When evidence is insufficient, the issue relies on visual details, real-time information, or expert judgment, AI should proactively inform users that the answer is unreliable, rather than pretending to be omniscient.

For ordinary people like you and me, swept forward by the tide of the times, the most valuable skill may simply be relearning how to say, “I don’t know.”

In an era where AI always provides answers, saying "I don't know" is not an admission of ignorance, but proof that judgment still exists.

Reference link:

[1] https://www.theregister.com/ai-and-ml/2026/07/19/using-ai-makes-people-less-likely-to-admit-they-dont-know-something/5274567

[2]https://arxiv.org/pdf/2607.13562

This article is from the WeChat official account "Quantum Bit," authored by Wen Ting.