Warning of Risks from Rapid AI Advancement
Need for a Framework Similar to Nuclear Arms Treaties

On June 4 (local time), Anthropic called for AI research labs to consider slowing down the pace of their development. The company pointed out that as AI advances too rapidly, it may soon reach a stage where it can improve itself without human intervention, which could pose significant societal risks.


On this day, Anthropic released internal data demonstrating performance improvements of its cutting-edge model in a blog post titled "When AI Builds Itself." The post was co-authored by Marina Favaro, head of Anthropic Institute—Anthropic’s internal research organization—and Jack Clark, co-founder and policy chief.

Anthropic logo. Photo by Reuters and Yonhap News Agency

Anthropic logo. Photo by Reuters and Yonhap News Agency

View original image

According to the post, AI models appear to be approaching the so-called "recursive self-improvement" stage, where they can enhance themselves without human involvement. Some in the AI industry believe this stage could signal risks and major societal upheaval. Anthropic warned that while recursive self-improvement has not yet occurred and is not inevitable, "it could arrive much faster than most organizations are prepared for."


Anthropic stated, "We believe it is desirable to have the option to slow down or temporarily pause the development of advanced AI so that social institutions and research can keep pace with technological progress." The company proposed an international consensus on slowing development, along with a verification system to ensure research labs comply. They emphasized that pausing or reducing the pace of development is only meaningful if widely followed, and suggested that a regime similar to the Intermediate-Range Nuclear Forces (INF) Treaty, which once drastically reduced nuclear missile deployment, is needed.


The company also noted, "The window of opportunity to examine this issue together is open right now, and people outside AI companies should also participate in this discussion."



Anthropic’s management, including CEO Dario Amodei, have warned in recent years about the impact of AI.

Anthropic: "AI Will Soon Advance on Its Own... Development Pace Must Slow Down" View original image

However, some have pointed out that Anthropic’s warnings may be part of a marketing strategy. For example, some claim that decisions such as restricting the release of "Claude Mythos" due to its alleged dangers are a way to showcase the performance of their products. Ethan Mollick, a professor at the Wharton School of the University of Pennsylvania, commented on social networking service X (formerly Twitter), "There's a mix of self-reflection and marketing elements, but there's also a lot of Anthropic's genuine beliefs about the near future of AI."


This content was produced with the assistance of AI translation services.

© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.

Today’s Briefing