Following Fake Documents Leads to Wrong Answers... AI Search and Reasoning Flaws Exposed [Reading Science]

Korea University Develops Attack Technique That Manipulates AI's Search and Reasoning Paths Through External Documents
Attack Succeeds in Up to 78% of Experiments, Revealing Security Vulnerabilities in AI Agents

A study has found that manipulated documents posted externally can alter an AI's search direction and even its final answer as it independently finds and analyzes information online. The finding has drawn attention as a new security threat to AI agents, since it shows that search and reasoning processes can be distorted step by step without infiltrating the AI system itself.

Korea University said on October 28 that a research team led by Professor Jang Buru of its Department of Computer Science developed an attack technique called “KidnapRAG” to manipulate AI search and reasoning processes and used it to identify security vulnerabilities in AI agents.

Unlike conventional attacks, AI can detect errors in manipulated information and return to normal reasoning. KidnapRAG also disrupts this error-correction process, leading the AI to continue making incorrect judgments. Photo provided by the research team.

Services in which AI searches for and analyzes information multiple times before generating an answer, such as the “Deep Research” features of ChatGPT and Gemini, have recently become more widespread. These technologies are based on “Agentic RAG,” an advanced form of retrieval-augmented generation (RAG), which enables AI to search external sources and use them in its answers.

The problem arises when the sources searched by AI contain information deliberately manipulated by an attacker. “Prompt injection” attacks, in which malicious instructions are inserted into external documents to change AI-generated answers, were already known. This study went a step further, showing that an attacker can steer the paths of repeated searches and reasoning themselves.

Manipulated Documents Surface With Each Search, Gradually Distorting AI Judgments

KidnapRAG, developed by the research team, works by posting manipulated documents externally without accessing the AI system itself. It steers the AI to conduct additional searches in a direction desired by the attacker based on the first document it retrieves, and is designed to make manipulated documents appear again in subsequent searches.

The intermediate steps showing what information the AI searches for and what judgments it makes can also be exploited. Based on these steps, an attacker can anticipate what the AI is likely to look for next and place documents tailored to that search stage. When this process is repeated, the AI may collect information and reach conclusions in the direction intended by the attacker, instead of finding an answer to the original question.

In the research team's experiments, KidnapRAG achieved an attack success rate of up to 78%. This was the highest rate recorded under the experimental conditions set by the researchers; it does not mean that attacks will succeed at the same rate across all AI services. The press release also did not confirm that any specific commercial service, such as ChatGPT or Gemini, had actually been attacked.

Research team photo. (From left) Jang Buru, professor in the Department of Computer Science at Korea University and corresponding author; Choi Chanwoo, integrated master's and doctoral program student and co-first author; Kim Euntae, integrated master's and doctoral program student and co-first author; and Lee Gyuho, master's student and co-first author. Provided by Korea University.

The study suggests that it is necessary not only to verify the reliability of individual documents retrieved by AI, but also to protect the entire process through which searches and reasoning unfold. The research team said it expects the findings from its analysis of the attack technique to inform the development of security technologies to detect and block manipulation of AI agents' search paths.

Professor Jang said, “This study shows that search and reasoning paths can be attacked under conditions close to those of real-world service environments, at a time when Agentic RAG is spreading rapidly. We hope the findings will be extended into defensive technologies that verify and protect the entire search and reasoning process, contributing to the development of safer and more reliable AI agents.”

The study was conducted jointly with KT Frontier Lab, and its findings were accepted for presentation at EMNLP 2026, an international conference on natural language processing taking place in Budapest, Hungary, from October 24 to 29.

This content was produced with the assistance of AI translation services.