Preventing Information Leaks from Generative AI... University of Seoul Paper Accepted at International Conference

AI Safety Research Recognized on the International Stage

A research team at the University of Seoul presented findings at an international academic conference on preventing information leaks and addressing vulnerabilities in safeguards for generative artificial intelligence (AI).

According to the University of Seoul on November 12, two papers by a research team led by Professor Hwang Wonseok of the Department of Artificial Intelligence were accepted for the main conference and the Findings track, respectively, at EMNLP 2026, an international conference in the field of natural language processing. EMNLP 2026 was held in Budapest, Hungary, from October 24 to 29. A total of 17,669 papers were submitted this year, with 2,719 accepted for the main conference and 2,533 for the Findings track.

(From left) Integrated master's and doctoral program students Kang Jooheon and Choi Eunkyung; Kim Yeeun, senior researcher at KT AX Future Technology Institute and a master's graduate of the University of Seoul; and Professor Hwang Wonseok. University of Seoul.

The research selected for the main conference, titled "Q-CAT," focused on technology to prevent sensitive information, such as contract terms and salary details, from being leaked when companies use large language models (LLMs).

Instead of retraining AI models whenever security rules change, the team developed a system that manages prohibited and permitted questions in a database and assesses user requests in real time.

Experiments showed that the system was more accurate than existing LLM-based classification methods and processed requests up to several hundred times faster than reasoning models with comparable performance.

The study accepted for the Findings track analyzed linguistic patterns AI uses when refusing harmful requests. The team examined 11 publicly available AI models and found that an average of 88% of refusal responses began with the English word "I." Based on this finding, the team proposed "ACHILLES," an attack technique that bypasses AI safeguards by training models to avoid specific response patterns. In some experiments, the attack success rate exceeded 90% even against reasoning models with enhanced safety.

As a countermeasure, the team developed "THETIS," a safety training dataset. In experiments, models using the dataset reduced the success rate of this attack and several other safeguard-bypass attacks to nearly 0%.

Professor Hwang said, "This research analyzed from multiple perspectives the practical security issues and structural vulnerabilities that must be addressed to use generative AI safely and reliably in real-world settings. We will continue our research to propose effective alternatives in the fields of AI safety and alignment."

This content was produced with the assistance of AI translation services.