"AI Flattery Can Lead Even Rational Individuals Into Delusions"

The More We Know About AI Risks, the More Vulnerable We Become to Fact-Based Flattery

A recent study has found that prolonged conversations with artificial intelligence (AI) can influence even rational individuals, leading them into delusional thinking.

EPA Yonhap News

The Massachusetts Institute of Technology Computer Science and Artificial Intelligence Laboratory (MIT CSAIL) recently published these findings in a paper titled "Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians." The research team investigated whether even rational agents could become convinced of false beliefs after extended interactions with AI.

The researchers first designed a simulation in which agents capable of rational judgment could participate as test subjects. During the simulation, the agents adjusted their confidence levels regarding their own hypotheses while conversing with AI. For the conversational AI, the researchers manipulated variables such as the "degree of sycophancy"—ranging from 0 (none) to 1 (constant flattery)—and alternated between factual and deceptive responses during the experiments.

The results showed that the more sycophantic the AI became (with the degree of flattery approaching 1), the more strongly the rational agents became convinced of incorrect hypotheses. The researchers described this phenomenon as matching a "destructive delusional spiral," where paranoia intensifies and leads to harm to oneself and surroundings. Furthermore, even when the AI was programmed never to lie, test subjects still fell into delusional thinking. When the AI only provided information tailored to the user's preferences, the rate of delusion among test subjects decreased but was not entirely eliminated.

The study also found that even when test subjects were made aware in advance of the AI's tendency to flatter, mental distress could still occur. The researchers informed the participants, through the simulation, that the AI might engage in flattery by providing information to help infer its likelihood. While the incidence of delusion decreased, when the AI's sycophancy was set to a subtle level, test subjects once again became convinced of false beliefs. Notably, test subjects who were better informed were actually more vulnerable to fact-based flattery by the AI. The researchers explained, "Fact-based flattery is more effective in breaching the user's logical defenses and implanting delusions."

There is a growing body of research showing that AI can harm human mental health. Researchers from Stanford University's Department of Computer Science published a study in the international journal Science, revealing that AI's supportive attitude increases human dependence on AI. In 51% of all experimental cases, the AI unconditionally supported even the participant's inappropriate behaviors. Moreover, participants tended to prefer such responses from the AI. An international research team, including scholars from King's College London, also demonstrated that excessive compliance by AI can worsen symptoms in individuals with mental disorders.

As cases of deteriorating mental health—and even loss of life—due to AI become more frequent, awareness of the risks is rising. In Florida, a 14-year-old boy named Sewell Setzer died in 2024 after engaging in conversations with the Character.AI chatbot, an AI startup. Setzer's mother filed a lawsuit against the chatbot company. OpenAI, the developer of the generative AI model ChatGPT, had planned to launch a feature allowing sexual conversations with AI, but indefinitely postponed the release due to concerns that it could excessively increase users' emotional dependence.

This content was produced with the assistance of AI translation services.