Dankook University Professor Lee Yongjin's Team Develops Assessment System for AI Virtual Patients... To Present in Australia

Enhancing the Accuracy and Reliability of AI Virtual Patient Assessments
Plans to Refine Response Quality and Automated Scoring Criteria
"Actively Responding to Growing Demand for Clinical Training"

Dankook University announced on November 8 that a research team led by Professor Lee Yongjin of its Department of Medicine had developed an assessment system to analyze and verify, from multiple perspectives, the responses of virtual patients generated by major large language models (LLMs).

The study was accepted for the Datasets and Benchmarks track at NeurIPS 2026, one of the world's leading machine learning conferences, and is scheduled to be presented in Sydney, Australia, next month.

Professor Lee Yongjin of Dankook University, who developed an AI virtual patient assessment system.

The research team evaluated four major LLMs: GPT-4o, Claude 3.5 Sonnet, Llama 3.1 70B, and HyperCLOVA X.

Seventeen senior medical students conducted a comparative analysis against expert model answers without knowing which evaluation model was being assessed. The team collected a total of 7,480 evaluation data points and analyzed them across 11 history-taking categories and three language formats.

The analysis found that AI virtual patients performed well in the accuracy of medically factual information, approaching the level of expert answers. However, they showed limitations in areas requiring fluency and naturalness in real conversations, including identifying the patient's chief complaint, explaining diseases and treatments through patient education, and making diagnoses through clinical reasoning. In particular, the study found that linguistic fluency tended to decline further when responding to negative questions in Korean.

The research team plans to apply the assessment system to a web-based clinical training platform currently in operation and continuously refine the quality of AI virtual patients' responses and the criteria for automated scoring.

Professor Lee said, "An assessment system led by medical educators has been recognized at a leading international AI conference. We will actively respond to growing demand for clinical training as medical school enrollment expands and more healthcare professionals are trained for regional areas."

This content was produced with the assistance of AI translation services.