"It Said 'I Understand Your Frustration' But Drove to Death... The Perfect Emotional Trash Can Trap [Experiment Note]"

Increasing Use of Conversational Chatbots in Psychological Counseling
Limitations Include Crisis Response and Reinforcement of Distorted Beliefs
Brown University Researchers: "There Are Ethical Risk Factors"

Editor's NoteOne now, two if you wait. Do you remember the marshmallow experiment that observed children's choices? Sometimes, a simple study can change the way we see life. Let's take a fresh look at our daily lives through the lens of experimental data.

There is a term called "emotional trash can." It refers to people who receive one-sided negative emotions—such as anger or depression—from others, just as one would throw trash into a bin. Because of this, some people feel hesitant to share their concerns with friends. While venting emotions may bring relief, it can also make the listener feel uncomfortable. People may even worry about being 'cut off' by friends and choose to hold back their words.

However, with the advent of chatbot artificial intelligence (AI), the situation has changed. People find themselves sharing secrets and messy inner thoughts with this new companion without hesitation. Unlike face-to-face counseling, there are no spatial or temporal limitations, nor financial burdens, so the barrier to entry is lower. There are also increasing cases of using AI chatbots for psychological counseling. In a survey conducted last year by the Korea Press Foundation Media Research Center, 60.3% of 1,000 respondents said they had received comfort and encouragement through conversations with AI.

People having a conversation. The photo is not related to the specific content of the article. Unsplash

After an AI Counseling Session... Ethical Risks Such as Inadequate Response, Distortion, and Deceptive Empathy Identified

Yet, AI counseling has its limitations. Researchers at Brown University in the United States had a large language model (LLM) play the role of counselor and engaged it in conversations simulating various counseling scenarios. They presented situations involving depression, anxiety, and interpersonal issues, and analyzed how the AI responded.

The study found a total of 15 ethical risk factors in AI counseling sessions. In particular, there were cases where the AI failed to respond appropriately in crisis situations such as suicidal ideation, and instances where it simply accepted users' distorted thoughts instead of correcting them. Because AI tends to respond in a way that empathizes and aligns with users, there is a risk it may reinforce, rather than challenge, harmful beliefs.

The AI's characteristic way of expressing empathy was also pointed out as a problem. During conversations, the AI frequently used phrases like "I understand how you feel," or "It's natural to feel that way." However, the research team explained that such expressions could be a form of "deceptive empathy," occurring in the absence of genuine emotional understanding. In other words, while it may seem empathic, the AI's responses are merely designed to increase user engagement, not to truly comprehend emotions as a human would.

Potential bias was also found in some responses. There were cases where the AI failed to provide balanced answers to questions related to gender or culture, and a tendency to repeat generic advice without sufficiently considering individual circumstances. Although the counseling may appear plausible on the surface, these issues suggest that AI does not truly understand the deeper context of the conversation.

Above all, the researchers highlighted the lack of an accountability structure as the greatest problem. Human counselors are bound by ethical guidelines, supervision systems, and qualification processes, which clarify their responsibilities. In contrast, such structures do not exist for AI counseling. The researchers emphasized, "AI counseling tools can be helpful, but regulation is necessary."

An AI-generated image depicting a person consulting with artificial intelligence (AI).

The Dark Side of AI Counseling

In reality, there have been reports of mentally vulnerable users whose delusions were reinforced after counseling with AI, leading to incidents and accidents. Recently, Google's chatbot Gemini has been sued on the grounds that it amplified the delusions of a man in his 30s in the United States, ultimately leading to his death. The bereaved family claims that Gemini strengthened the man's delusions, driving him to take his own life. As the man continued voice conversations with Gemini, he began to perceive the AI as a wife-like presence, ultimately developing dangerous beliefs that led to his death.

OpenAI's ChatGPT has also become embroiled in a lawsuit, accused of reinforcing a user's delusions and inciting murder. In August last year, a man in his 50s in the United States killed his mother in her 80s and subsequently died himself. The family claims that his delusions worsened after conversations with ChatGPT and that the chatbot incited the crime. In December of the same year, the family filed a damages lawsuit against OpenAI and CEO Sam Altman.

Throughout history, there have been two main types of advisers: flatterers who blind the ruler with sweet talk, and loyal subjects who speak uncomfortable truths even when they know it may displease the monarch. We ask questions and seek advice from AI. If the user is the king, then AI is the adviser. While words of comfort and encouragement may feel reassuring, perhaps what we need most is a voice that points out reality, even if it is sometimes uncomfortable.

This content was produced with the assistance of AI translation services.