[Science Scope] AI Writes the Papers, AI Reviews Them... The Trust in Science Is Faltering

Over 140,000 Fake References Created by AI:
Paper Output Surges, But Verification Stalls

AI Learns from Data; Science Relies on Trust in People

Editor's Note'Science Scope' is a column that, like the word 'scope,' zooms in on specific science and technology phenomena to offer in-depth analysis of their significance and future.

References and citations created by generative artificial intelligence (AI) that do not actually exist are now starting to appear in scientific papers. As the era in which AI writes and helps review manuscripts arrives, the scientific community is concerned that the trust system—maintained for more than 300 years, whereby people write and people verify—could be shaken.

Newton conceived the law of universal gravitation by observing a falling apple. Einstein established the theory of relativity through thought experiments. If so, what does generative AI observe when it engages in science?

ChatGPT Generated Image

AI does not observe nature. Instead, it learns from tens of millions of papers and datasets to generate the most plausible answers. Although it was developed as a tool to aid research, in just two to three years, AI has transformed the ecosystems of manuscript writing, review, and publication. The key question for the scientific community is no longer how well AI can write papers. Instead, it is how we can trust science in the AI era.

AI Has Already Entered the Laboratory

Generative AI is no longer an experimental tool for a handful of researchers. The international scientific publishing sector is already developing new rules premised on AI usage.

Springer Nature, the world's largest international scientific publisher, which publishes thousands of journals including Nature, permits the use of AI for tasks such as proofreading and translation but does not recognize AI as a paper author. This is because AI cannot take responsibility for research outcomes.

Elsevier, another global leader in scientific information and publishing, also requires transparent disclosure of generative AI usage. The company has set out the principle that ultimate responsibility for the research content and conclusions lies with the researchers. In particular, as references generated by AI may be false or inaccurate, Elsevier emphasizes the necessity for authors to personally verify all such citations.

Leading journals such as Nature and Science also do not prohibit the use of AI per se. Instead, they have repeatedly warned that 'hallucinations'—the fabrication of nonexistent references or plausible-yet-false statements by AI—can undermine the credibility of scientific research.

The use of AI is evident in research outcomes as well. In a study published in 2025 in the international journal Science Advances, Dr. Dmitry Kobak's research group at the University of Tübingen in Germany analyzed 15.1 million biomedical paper abstracts listed in the PubMed database from 2010 to 2024. The researchers found that the usage of certain expressions and styles increased sharply after ChatGPT was released, and they estimated that at least 13.5% of biomedical abstracts published in 2024 may have been assisted by large language models (LLMs).

The problem has grown much more severe in just one year. In a study released in 2026, Zhao Zhenwei, a researcher at the Department of Information Science at Cornell University, and Professor Inyen's team analyzed the world's leading online platforms where researchers first publish and share their preprints. The collaboration included Paul Ginsparg, founder of the world’s largest preprint server, arXiv.

The research team analyzed around 2.5 million papers and 111 million references published on platforms such as arXiv (physics), bioRxiv (life sciences), SSRN (social sciences), and PubMed Central (biomedical sciences, run by the U.S. National Institutes of Health). They found that in 2025 alone, at least 146,932 hallucinated citations—references that do not exist—may have been included.

This means that fabricated references generated by AI are already infiltrating real academic literature. AI is now not only writing papers, but also generating the references that serve as the foundation of scientific knowledge.

The Issue Is Not AI, but Verification

AI has replaced tasks like English proofreading, literature searches, and abstract writing, significantly reducing the time needed to draft manuscripts and enhancing research productivity. However, the authenticity of papers, reliability of data, appropriateness of statistical analysis, and accuracy of references still require human verification.

This is where the scientific community’s concerns begin. Nature has analyzed this year that fabricated references created by AI are being included in papers across various academic fields, and that existing peer review and editorial processes are not sufficient to fully filter them out.

Paper retractions are also increasing rapidly. According to Retraction Watch, a global paper retraction monitoring group, over 10,000 papers were retracted worldwide in 2023, marking a record high.

The bigger issue is the limitations of verification systems. At Springer Nature, the world’s largest academic publisher, the number of manuscript submissions surged to more than 3.1 million in the past year alone. However, the peer review system needed to verify these papers cannot keep pace, causing a growing burden on reviewers.

As Humans Became Scarce, AI Began to Review

Because the pace of paper production now outstrips that of verification, it is the peer review system—the core review process of science—that has first hit its limits.

Peer review has long suffered from a chronic shortage of reviewers because it relies heavily on voluntary and largely unpaid participation. To make matters worse, as the number of submissions explodes as described earlier, there are far too few expert researchers available to review and verify all manuscripts. As a result, review periods are becoming indefinitely long and researcher fatigue is reaching its peak.

Eventually, as human verification reached its limits, AI started to be used in the review process as well. Some academic publishers have implemented systems that use AI for preliminary checks on statistical errors, plagiarism, references, and image manipulation. Recently, there has also been an increase in cases where AI drafts peer review comments and researchers revise and supplement them. There are now substantive discussions about the extent to which AI can or should be used in the review process.

This has encountered considerable resistance. For example, Timothée Poisot, an ecologist at the University of Montreal in Canada, wrote on his blog after receiving reviewer comments that appeared to have been generated by AI, stating, "ChatGPT is not my peer." He added, "I submitted my paper expecting feedback from fellow researchers, not from AI, and the moment that expectation is broken, the social contract of peer review collapses as well."

The Even Greater Problem: Verification Speed

In the scientific community, networks that produce or mass-produce papers using paid writing services or manipulated data and images are referred to as "paper mills." In recent years, this issue has become one of the most concerning research ethics problems facing international scholarly publishing.

There are concerns that generative AI could further boost the productivity of such "paper mills." In the past, producing even a single paper required much time and labor. But by learning from existing papers, generative AI can now produce manuscripts, illustrations, graphs, and references in a very short time. Even without generating new experimental data, it has become much easier to combine previous research to produce seemingly high-quality papers.

Of course, AI did not create the phenomenon of paper mills. But AI has enabled paper mills to produce more papers with less time and cost, and experts note that the verification burden on scholarly publishers is growing commensurately.

Ivan Oransky, co-founder of Retraction Watch, which operates the world's largest database of retracted papers, has long stated that the increase in retracted papers is "science correcting itself." Still, concerns are mounting that when the pace of paper production outstrips that of verification, the self-correcting mechanism of science could ultimately reach its limits.

Korea Is Also Setting Research Ethics Guidelines for AI

In Korea as well, standards for the use of generative AI are being established in earnest. In June, the National Research Foundation of Korea published "Research Ethics Guidelines for Generative AI for University Researchers." The key points are that AI cannot be included as an author, and any use of generative AI must be transparently disclosed. Researchers bear ultimate responsibility for the accuracy of research outcomes and for upholding research ethics, and the principle is that the confidentiality of reviewed manuscripts must not be compromised through AI usage in the review process.

The European Union (EU) also recently stated in its guidelines for generative AI in research that "AI is only a tool to support the expertise of researchers, not a substitute for their judgment and responsibility." This makes explicit that, even in the AI era, accountability for research ultimately lies with people.

The international academic journal Nature warned last April that 'hallucinated citations' generated by generative AI are being included in actual papers, threatening the credibility of scientific literature. Provided by the Nature website

Byungmook Won, Professor in the Department of Advanced Materials Science and Engineering at Sungkyunkwan University, stated, "Scientific honor and evaluation can only stand on verifiable research outcomes and rigorous research ethics. As generative AI makes it easier to write papers and organize data, the risks of false citations, unclear sources, and shifting responsibility are also rising." He additionally emphasized, "Only by strengthening openness about AI usage, preserving original data, and reinforcing independent verification procedures can we safeguard trust in science."

Thanks to AI, papers can now be written more quickly and, going forward, more papers will be produced. Yet science does not advance merely by the quantity of publications. New knowledge is only established as science after it passes through observation, experimentation, and verification by other researchers.

AI learns from data. But science is a system that relies on trust in people, beyond just data. What must ultimately be protected in the age of AI is not the algorithm, but trust itself.

This content was produced with the assistance of AI translation services.