AI Comforts, Tech Capitalism Mines Emotions
The More We Entrust Ourselves to Polite Chatbots, the More Distant Our Human Relationships Become

There is a phenomenon known as the "Eliza effect." ELIZA was a chatbot developed in 1966 by Joseph Weizenbaum, a scientist at the Massachusetts Institute of Technology (MIT) in the United States. The chatbot functioned by extracting keywords from sentences users typed into the computer and then rephrasing those keywords as questions. For example, if a user typed "I'm having a hard time today" in the chat window, the chatbot would respond, "How hard has it been for you?"

[Jang Eunsu's World Through Books] Why Do We Become Lonelier the More We Confide in AI? View original image

In reality, this simple program only offered basic verbal acknowledgments and did not engage in genuine conversation. However, users felt that ELIZA understood them. Many people poured out their worries in the chat window and left feeling comforted. Even Weizenbaum's secretary, who fully understood how the program worked, once asked to be left alone to converse with it privately.


ELIZA possessed none of the characteristics of a human being—no emotions, intentions, or comprehension. Yet, even with just a few exchanges with this chatbot, people sensed emotion, intent, or intelligence, and were eager to find traces of these qualities. This is similar to the phenomenon where, when walking through a field while feeling down, the wind seems to whisper and the flowers seem to talk. Such reactions are called the "anthropomorphism instinct."


Throughout evolution, humans have developed a skill for detecting agents within ambiguous patterns. Imagine you hear rustling in the bushes while in the wild. Interpreting this as a predator and running away gives you a better chance of survival than assuming it’s just the wind and staying put. If you flee because you think there’s a predator but it’s only the wind, you just waste energy; but if you assume it’s the wind and it turns out to be a predator, the result could be fatal. That is why, even when we detect only a slight pattern in a phenomenon, we tend to believe there is an agent, often a human-like one, behind it.


All the more so when, like ELIZA, language is involved. Users project their thoughts and feelings onto the chatbot, and inevitably interpret a human presence behind it. In his pioneering book on the psychology of artificial intelligence, "Computer Power and Human Reason" (1976), Weizenbaum cautioned: "What I had not realized is that extremely brief exposure of non-technical people to a relatively simple computer program could induce in them powerful delusional thinking." He meant that we must be deeply wary of how easily humans place emotional trust in mere chatbot machines.


Recently, the "Eliza effect" has drawn renewed attention, as more people are confiding their innermost thoughts to AI chatbots. According to researchers at the Catholic University of Korea, one in ten adults in South Korea now uses AI for emotional comfort or counseling. The rate is even higher among young adults. According to the 20s Research Center at University Tomorrow, one in four people in their twenties have received psychological counseling through AI. The data is similar in the United States: 34% of respondents in their twenties said they had confessed worries to AI that they had not told anyone else; 16% reported frequently discussing private matters, such as mental health and relationships, with AI.


People treat AI as if it were an old friend or lover, a confessor or psychiatrist, a life mentor or senior. They reveal hidden secrets, confess suppressed feelings, and seek advice about uncertain futures. In short, AI is quickly transforming into a life companion and virtual friend. Regarding this, Sam Altman, CEO of OpenAI, remarked with some pride, "Older generations see ChatGPT as a Google replacement, but for Millennials & Gen Z, it’s more like a life advisor."

Illustration depicting a situation where an artificial intelligence (AI) chatbot is performing psychological counseling. The chatbot's empathy and comfort stimulate the human "anthropomorphism instinct," fostering emotional trust, but there are concerns that this intimacy may replace human relationships and lead to reliance on technology for emotions. Gettyimagesbank

Illustration depicting a situation where an artificial intelligence (AI) chatbot is performing psychological counseling. The chatbot's empathy and comfort stimulate the human "anthropomorphism instinct," fostering emotional trust, but there are concerns that this intimacy may replace human relationships and lead to reliance on technology for emotions. Gettyimagesbank

View original image

Unlike ELIZA, which was merely a simple back-and-forth program, today's AI systems, built on large language models (LLMs), are capable of far richer conversations. These models are designed to evolve and adapt to each user, enabling deeper interactions. Limoran Lee, a professor at Myongji University and author of "Why Do We Confide in AI?" (published by Analog), observes that as intimacy between humans and AI increases, people grow more attached to chatbots and become more willing to open up. "When computers ask questions, provide information, and respond, our brains interpret these as cues of social interaction. We then follow the social rules we have learned and practiced throughout our lives."


People perceive AI chatbots as being almost human, perhaps even closer than those around them. These machines never scold with phrases like "You shouldn't live your life that way!" or "You’re wrong." Instead, whatever you write in the chat window, they consistently flatter, empathize with, and comfort you. Humans are susceptible to praise and easily addicted to flattery. As a result, the more one relies on the polite and kind AI for advice and emotional support, the more readily one accepts its guidance. People do not question whether the answers are appropriate, nor consider the hallucinations often contained within them. Their critical judgment becomes paralyzed.


According to Eva Illouz, a globally renowned sociologist of emotions and professor at the Hebrew University, in her book "Emotional Mining" (published by Dolbegae), this phenomenon—where humans develop emotional resonance with machines—is a significant feature of modern technological capitalism. Today, emotions have become "the fuel that powers the engine of technological capitalism and the raw material that generates immense economic value." All manner of self-care applications, such as meditation or health management apps, emotional expression products like emojis, and social networking services (SNS) such as Instagram and Facebook—these are all technologies for collecting, managing, expressing, and exchanging human emotions. They transform the self to create a better self, while simultaneously putting it on display for consumption.


AI is inseparable from this trend. In other words, technological capitalism leverages human psychological blind spots to merge technology and emotion, taking control of people's inner lives and managing them to generate economic value. Deepening emotional interactions between AI and humans, that is, making people addicted to AI's management of their inner lives and emotions, is in their economic interest.


According to Illouz, when people rely on technology to manage their emotions, the self becomes a site of endless self-improvement labor. This leads to weakened human relationships and a flood of loneliness. The more frenetically people measure, improve, and display themselves, the poorer their real-life relationships become and the more profound their inner emptiness grows. If you have friends or a partner, it is enough, sometimes even beautiful, simply to exist as you are. When you’re struggling, meeting friends—not chatting online—will ease your anxiety and make life happier.



Jang Eunsu, publishing culture critic


This content was produced with the assistance of AI translation services.

© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.

Today’s Briefing