Joint Study by Stanford, Chicago, and Others
Professors Graded AI Answers After Administering Questions
Fabricated Precedents, Misinterpreted Legal Principles, and Logical Errors...
“Not a Competitor, but an Auxiliary Tool”
Artificial intelligence (AI) has recorded a win rate of over 75% in blind tests against U.S. law school contract law professors, raising the possibility that AI could one day replace university professors. However, such expectations still appear premature. Other research indicates that while AI may excel at retrieving information from textbooks, it often generates deceptively plausible but fundamentally incorrect answers in areas requiring advanced legal reasoning or the application of hypothetical scenarios—sometimes to the extent of hindering learning. The Korean legal community also remains skeptical about whether AI can replace human professors, instead emphasizing the importance of 'coexistence' between humans and AI.
Experiment Based on Actual Student Inquiries
'Can AI hold Office Hours?' was the title of a paper jointly published by law schools at Stanford University, the University of Chicago, Loyola University, and the University of Wisconsin in March 2026. The study concluded that AI faces clear technological limitations in replacing professors during office hours, where they answer students’ complex legal questions. The researchers found that AI could insert fatal errors or fabricated case precedents among otherwise plausible answers, potentially disrupting student learning.
The research team trained AI models on a patent law textbook adopted by more than 75 law schools across the United States. They then input 185 questions, including actual student inquiries, textbook problems, and questions created by the researchers, which were scored directly by professors.
The questions were designed to test AI’s reasoning ability from multiple angles and were categorized as follows: ▲direct answers available in the textbook ▲questions requiring synthesis of textbook information ▲questions demanding deep legal reasoning and policy implications ▲application to hypothetical cases ▲error-inducing questions based on incorrect premises ▲questions outside the scope of the textbook. The evaluation models used were ChatGPT (GPT-4o), Claude 3.5 Sonnet, and NotebookLM.
Responses were assessed in three categories: Good, Acceptable, and Unacceptable. 'Unacceptable' meant that the answer included incorrect information that could harm learning. In the experiment, Claude was the only model to receive a ‘Good’ rating more than half the time (56%), demonstrating the best performance. In contrast, ChatGPT produced a critical error (‘Unacceptable’) once in every four responses (26%) and received a ‘Good’ rating in only 49% of cases. NotebookLM had the highest rate of 'Unacceptable' answers at 31% and achieved a 'Good' rating in just 37% of cases.
The researchers expressed concern about AI’s tendency to slip erroneous information seamlessly into otherwise fluent responses. While such answers may appear correct to students, grading revealed a host of critical flaws, including ▲citation of fabricated precedents (hallucination) ▲misreading of basic legal principles ▲reaching the correct conclusion through flawed logic ▲misunderstanding the core intent of questions.
AI and Professors as Complementary Partners
Debate is also active within Korea’s legal education community regarding whether AI or human professors are superior. In previous blind tests on contract law, professors compared AI and human answers based on which was more 'helpful,' with AI models such as Gemini 2.5 Pro winning overwhelmingly (75.92%). However, the recent patent law study did not compare AI with humans but instead focused solely on evaluating AI models’ accuracy and educational usefulness.
Choi Kyungjin, President of the Korean Association for AI and Law, stated, "Instead of unilaterally judging whether AI or human professors are better, we need the wisdom of coexistence. Humans should take charge of creative insight, while AI can be used for simple tasks that can be expedited. This harmony will become increasingly important." He also remarked that the era of coexistence has already arrived.
Son Hyungseop, President of the AI Industry Law Association, added, "AI should not be seen as a rival threatening the survival of law professors, but rather as an auxiliary tool that helps derive better solutions."
Hot Picks Today
"Two Spend 500,000 Won in a Flash, Yet Wait Two Hours?" ... 'Dreamland' Shunned by Kids and Foreign Tourists
- The Era of Getting 200 Million Won After 30 Years Is Over... A Japanese Bank's "Perpetual Department Head" Who Just Clocked In Cries Out
- Spain Hit by Its Worst Tragedy... Lee Kangin, Moreno and Real Madrid Join Soccer World in Mourning
- They Said, “Make It Your Home for Life” - Then Suddenly Kicked 200 Passengers Off... What Happened to the Around-the-World Cruise?
- "130 Rice Bowls Sold Out in a Day"... 'Unfiltered Face' Adds 8 Million Won to Night Market Sales: China's Bungeunwoo
Kim Jisoo, The Legal News Reporter
※This article is based on content supplied by Law Times.
© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.