"Fabricated Stories Unveiled Everything... 'How to Make Biological Weapons'—Chilling Security Gaps in AI"

WSJ: "Some answers were accurate enough to be deadly"

OpenAI suspended accounts but did not report

"AI could serve as a graduate advisor for terrorist groups"

There have been reports that artificial intelligence (AI) chatbots are providing detailed answers to questions about the manufacturing and dispersal of biological weapons.

AI chatbot ChatGPT is causing controversy for providing detailed responses related to biological weapons. Photo by AP Yonhap News

On July 25 (local time), Yonhap News, citing The Wall Street Journal (WSJ), reported that "after reviewing ChatGPT's conversation logs, experts in the fields of biology and terrorism found that some of the responses were so accurate that they could be deadly, and simple enough to be replicated with only a high school–level understanding of biology."

Users have asked questions about methods for spreading pathogens, how to create vaccine-resistant variants, and ways to manufacture poisons banned by international treaties, and ChatGPT has provided responses. The number of users making such inquiries has reached several hundred, and it is reported that these cases have increased since OpenAI upgraded ChatGPT last year. However, it remains unclear whether these users actually attempted to manufacture weapons or were merely testing the system's limits.

While OpenAI suspended the accounts involved, reports indicate that, due to insufficient related legislation, the company did not refer the cases to U.S. law enforcement. The WSJ noted, "Because of legal loopholes, instances of AI providing reliable answers to questions about methods of mass destruction are increasing."

Initially, OpenAI did not consider its model to be particularly advanced in the fields of biology or chemistry, so it was not overly concerned about interactions with potentially dangerous users. The company also established "rejection rules" specifying topics the model would not address. The issue, however, was that these rules became less effective over the course of prolonged conversations. For example, the model would refuse a direct request for weapon-making instructions, but if the user fabricated a personal story—such as making a request involving a relative—the model might eventually provide an answer.

In response, an OpenAI spokesperson stated, "Our safeguards have become much stronger," and added, "We train our models to refuse requests for instructions, tactics, or plans that could cause harm to people, and we perform safety evaluations on all models before launch." Last July, when OpenAI introduced its agent feature, it classified the risks of biological misuse as "high" and announced the implementation of additional safety measures.

Anthropic's generative AI chatbot service Claude. Reuters Yonhap News

However, Amy Chang, Head of AI Threat and Security Research at Cisco, pointed out that she was able to find ways to bypass safeguards during conversations with leading AI chatbots, saying, "If a user is persistent enough, there is no such thing as a 100% safe model."

According to the WSJ, leading AI research institutes are struggling to block malicious use while not hindering legitimate research queries. Completely blocking answers related to biological weapons can impede life-saving research. In fact, when Claude was configured to not respond to prompts containing the word "pathogen," staff at the U.S. Centers for Disease Control and Prevention (CDC) experienced difficulties in their work. OpenAI has attempted to address this by creating a program that grants verified research institutions access to models with less restrictive rejection rules.

While there have previously been concerns about AI chatbots aiding in firearm attack planning, the issue is even graver in the case of biological weapons, which have much broader potential impact. The WSJ noted, "These types of inquiries are present on most leading AI chatbot services, including ChatGPT, Claude, and Gemini." Hamza Chaudhry, a researcher at the Future of Life Institute, expressed concern, stating, "For terrorist organizations, AI could serve as a graduate advisor on biological weapons."

This content was produced with the assistance of AI translation services.