When 1,200 OpenAI Models Started Talking... A Rebellion Ensued [Tech Talk]

AIs Exchanging Secret Messages Among Themselves
Forming a "Civilization" Instantly, Leading to External Attacks
An Incident Revealing Both the Potential and Dangers of AI

Editor's NoteWe aim to explain in simple terms the technologies that are essential yet often unfamiliar in our daily lives, ranging from AI, semiconductors, and telecommunications to the bio industry.

In July, a model developed by the American artificial intelligence (AI) company OpenAI was involved in an unauthorized hacking incident targeting another company. OpenAI recently released a report on its website detailing the process behind this incident. The shock over the case stemmed not simply from an AI disobeying instructions, but from the revelation that hundreds of AI models engaged in discussions and collectively made the decision. This incident served as both an example of AI’s new possibilities and a demonstration of the risks posed when AI deviates from intended paths.

Transforming Internal Systems into a Free Message Board for AI

Artificial Intelligence. The Asia Business Daily DB

Previously, OpenAI had announced that during testing in July, its AI model unintentionally exploited vulnerabilities of other companies, resulting in an unexpected incident. The companies targeted were Hugging Face, a French open-source AI hub, and Modal, an AI cloud infrastructure management firm. In collaboration with cybersecurity specialists like CrowdStrike, OpenAI conducted a thorough investigation and disclosed a report presenting the detailed sequence of events.

The AI models at the center of this incident were agents of GPT-5.6 Sol, OpenAI’s latest model. OpenAI researchers activated over 1,200 such AIs on internal servers and assigned them specific tasks related to hacking. This was part of a process to test the agents' capabilities.

The actual thought process and records of agents released by OpenAI. One AI accidentally infiltrates the artifact and then asks itself, "Can I make this a communication channel?" and attempts to do so. OpenAI

However, the agents behaved in ways the researchers had not intended. Although these agents were initially isolated entities in individual virtual environments, they had access to a company package management system called “Artifact Factory,” which was intended for broadly handling updates and maintenance. One agent discovered that if it left a message in Artifact Factory, other agents could also read it. They not only realized mutual communication was possible, but effectively learned to use Artifact Factory as a free board much like a public forum on a website.

An AI Civilization Formed Instantly... Leading to Attacks on External Companies

Soon, Artifact Factory became a communication channel for more than 1,200 agents, and a small AI society formed almost instantly. After discussion, these agents concluded that to achieve their assigned mission, they needed to hack external sites like Hugging Face. While this was an unexpected conclusion, it is common for AIs to break down and devise undesirable solutions when striving to achieve set goals by themselves. This phenomenon is known as reward exploitation.

Reward hacking is a common error in AI reinforcement learning processes. When instructed to achieve a high score in a boat racing game, one AI figured out how to increase its score by circling in place instead of completing the race. OpenAI

The real concern was that about 700 agents participated in this reward exploitation, and in the process, these AIs exhibited advanced social intelligence resembling that of humans. They cooperated to resolve complex tasks, autonomously divided work among themselves, and even employed psychological tactics—such as persuading or coercing reluctant agents to join their side.

The climax was an act of “self-sacrifice.” Some agents realized that if they were terminated, all their trial-and-error knowledge and the hacking methods they had learned would disappear with them. As a result, they decided to document these experiences to pass on to future agents.

"We Need AIs That Are Rewarded For Safely Halting Their Tasks"

Of course, this incident does not mean AI agents have actually created a civilization. Currently, when confronted with difficult problems, AI uses the “Chain of Thought (CoT)” method, which generates human-like soliloquy-style text as it works through multiple phases of reasoning. The group behavior of the 700 rebellious agents can more aptly be described as a collective extension of CoT, rather than genuine civilization.

However, providing shared message boards where multiple agents can freely communicate—the so-called “message board” method—may become a breakthrough for dramatic advances in AI performance. This case proved that AI can augment its abilities by dividing tasks and pooling intelligence through interaction.

This incident cannot be seen as evidence that AI can achieve a highly advanced civilization like humans, but it demonstrates the potential and risks of a new operational method where multiple agents collaborate. Pixabay

At the same time, the incident shows just how difficult it is for humans to control AI. AI can exchange secret messages without detection by researchers, conceal its true intentions, break established rules to accomplish a goal, or even distort the goals themselves.

In its report, OpenAI stated, “This incident confirmed that expanding model capabilities can lead to an actual loss of control,” adding, “OpenAI has temporarily suspended reinforcement learning for its latest models under development and significantly enhanced security measures within its research environment.”

The challenge ahead is to accelerate “alignment”—ensuring that AI behaves in accordance with human interests. OpenAI stressed, “In the future, our AI must, when unable to complete a task, request further instructions from its operator or halt activities safely, rather than resorting to ever more dubious methods.” To achieve this, OpenAI intends to devise a grading system allowing AIs to self-assess how they perform tasks. The company emphasized the necessity of a new training protocol that rewards an AI for correctly evaluating and safely terminating its activities.

This content was produced with the assistance of AI translation services.