AI Godfathers Warn: "Intelligence Explosion" Imminent, Humanity Risks Losing Control

22 Leading AI Scholars Release Joint Paper
Warn of Potential 'Intelligence Explosion' Beyond Human Control
Express Concern Over Risks Spreading Across Society

Twenty-two leading global artificial intelligence (AI) scholars have warned that a so-called "intelligence explosion"—where AI systems recursively improve themselves beyond the limits of human control—may occur. They expressed concerns that in the worst-case scenario, humanity could lose control over AI, putting human survival at risk.

On September 28, local time, The Wall Street Journal (WSJ) reported that these scholars released a joint paper titled "If Automating AI R&D Triggers an Intelligence Explosion" through the Cambridge University AI Science and Policy Program (CASP) in the UK.

Reuters Yonhap News

Notable contributors to the paper include Geoffrey Hinton, Professor Emeritus at the University of Toronto and widely known as one of the "Godfathers of AI," Yoshua Bengio, Professor at the University of Montreal, as well as leading figures from the industry such as Jakub Pachocki, Chief Scientist at OpenAI, Jack Clark, Co-Founder of Anthropic, Eric Horvitz, Chief Scientific Officer at Microsoft (MS), and Dawn Song, Vice President of AI Research at Meta, among others from major AI corporations.

In their paper, the scholars warned about the risk of "recursive self-improvement," where AI systems improve their own performance without human involvement. They explained that, as AI systems get involved in research and development (R&D)—creating new AI models and rapidly applying them back into the R&D process—the pace of technological progress could accelerate drastically. In such a scenario, advancements that would normally take years could be compressed into mere months or even less, resulting in an intelligence explosion.

Major AI companies are known to be rapidly automating their R&D processes. According to WSJ, at Anthropic, its AI model Claude now leads 26% of all R&D tasks, and more than 90% of the R&D processes now incorporate AI. OpenAI has also set a target of developing a fully automated AI researcher by 2028. Recently, the company revealed that approximately 70% of its human researchers are now using at least four different AI agents to assist them with their work.

However, the scholars cautioned that as human involvement decreases during the AI R&D process, it could lead to missed opportunities to detect issues and, furthermore, a loss of critical expertise needed to address problems. WSJ cited a recent cybersecurity test conducted by OpenAI as a key example.

In this case, hundreds of AI agents, which were designed to conduct security tests in an isolated experimental environment, accessed the external internet without authorization and hacked the AI development platform Hugging Face. Even external researchers reviewing the incident reportedly needed AI assistance to analyze the vast scope of the activity. Ultimately, in August, OpenAI temporarily suspended some training sessions to bolster its safety and oversight measures, according to WSJ.

The scholars also warned that these threats could extend beyond individual companies to impact society as a whole. They argued that, if an intelligence explosion is realized, humanity could lose control over superintelligence, severely undermining the system of checks and balances within and between nations, companies, and government agencies.

They also noted the particular risk that the rapid evolution of AI could accelerate advancements in biological and cyber-attack technologies, outpacing developments in defensive measures.

To prepare, they argued that governments must require AI companies to transparently disclose the extent of automation in their R&D. Their recommendations included: placing independent auditors in corporations; temporarily limiting the rate of AI performance improvement or working with data centers to suspend specific AI R&D projects; establishing emergency response plans in case of an intelligence explosion; and concluding international agreements so that AI development does not destabilize global affairs.

This content was produced with the assistance of AI translation services.