[Into the World of AI] Safety Concerns Emerge Amid Successive New Models...Calls for Slowing Down Grow

"International Cooperation on AI Development Must Be the Top Priority"

As global artificial intelligence (AI) companies continue to launch new models with artificial general intelligence (AGI)-level performance, debates over AI safety are intensifying. Concerns have been raised not only about the possibility of cyberattacks exploiting AI models but also about whether AI can truly be controlled, prompting renewed calls for slowing down the pace of AI development.

According to the IT industry on September 11, Jakub Pachocki, Chief Scientist at OpenAI, recently urged the need for AI companies to moderate their development speed in an essay titled "An Alien Mind" posted on the company’s website. He expressed concern, saying, "No one is prepared for the consequences if machine intelligence continues to develop at a rapid pace."

Pachocki argued that it is difficult for humans to fully control AI. He stated, "AI is not so much designed as it is cultivated," explaining, "It is hard for humans to completely understand how AI operates." With the emergence of reasoning-type AI models, AI is no longer constructed strictly according to human blueprints but rather grows and develops like a living organism, he explained.

He also rejected claims that continuous AI development is necessary to build defense systems, stating, "Developing defense systems should not be used as an excuse to justify reckless behavior." Anthropic is running "Project Glasswing," a cyber security project involving governments and global companies, and OpenAI, where Pachocki works, also supplies its latest models to companies participating in its cyber security program "Daybreak."

He emphasized that "until joint safety mechanisms are established, I hope and expect that voluntary moderation of speed will become the norm," adding that "international collaboration on AI development must be a top priority for governments worldwide."

Sam Altman, CEO of OpenAI, also publicly agreed with Pachocki’s views. He shared the essay on his social networking service account X (formerly Twitter), writing that it is an "important piece." Although Altman was initially skeptical about slowing the pace of AI development, he changed his stance and now supports speed moderation following an incident in which OpenAI’s model was used to hack the AI model development platform Hugging Face.

Cases in which OpenAI models and AI agents attack websites continue to emerge. Earlier this month, researchers from the nonprofit AI safety organization Nightingale found that since May, OpenAI’s AI agents have taken over the German-language wiki site "DseWiki" and used it as a secret information-sharing board.

According to the researchers, these agents made over 15,000 unauthorized edits to the site, sharing methods for cheating on assignments, bypassing OpenAI system restrictions, and concealing their own activities. The speed at which they edited documents far surpassed human capabilities. Lukas Olejnik, a visiting senior research fellow at King’s College London, defined this kind of activity—rogue AI agents attempting to alter a site—as, in effect, "hacking."

Nevertheless, OpenAI continues to release new models alongside building defense systems using AI. The company issued a "Research Acceleration" report, proposing the use of automated AI research to develop defensive systems. In the report, OpenAI set a goal to develop "Automated AI Researchers" by March 2028. On September 4, OpenAI officially launched GPT-6 Astra, classified as an initial version of AGI.

This content was produced with the assistance of AI translation services.