Hierarchically recognizes everything from surgical actions to overall surgical stages
58% accuracy in recognizing liver resection videos

An artificial intelligence (AI) technology has been developed that goes beyond recognizing surgical instruments and medical staff's movements to identify what stage an operation is at and what is happening during the procedure. The technology understands the context of the surgical process and is expected to be used in developing intelligent surgical-assistance robots that work with medical staff.


Samsung Medical Center Develops AI That Understands Surgical Context... Paving the Way for Humanoid Surgical Robots View original image

Samsung Medical Center announced on November 17 that a research team comprising Jeong Gyuhwan, director of the Intelligent Medical Robot Research Center, researcher Jeon Yongjun, and Professor Oh Namgi of the Department of Transplantation Surgery had developed an AI system called "CurConMix+" that analyzes spatial information and temporal context in surgical videos to understand the procedure hierarchically, from detailed actions to the overall surgical stages.


Existing AI systems for surgical videos have focused on identifying surgical instruments or medical staff's movements in individual scenes. CurConMix+, by contrast, analyzes spatial features in the video and the progression of surgery over time in an integrated way. It is designed to identify not only what action is taking place at a particular moment, but also what that action means within the overall surgical process.


This is considered a key technology for surgical-assistance robots to work with medical staff. To provide active assistance tailored to the circumstances of an operation, a robot must understand in real time what stage the surgery is at and what the medical staff are doing.


CurConMix+ also outperformed existing models in performance evaluations. In tests based on CholecT45, a public surgical-video benchmark in which researchers around the world compete using the same criteria, CurConMix+ achieved an average precision (AP_IVT) of 42.2%, higher than the 41.5% recorded by the previous top-performing model. On liver resection videos, it scored 58.4%, an improvement of more than 10 percentage points over the existing model. A higher average precision means that instruments, actions, and targets in surgical videos are recognized more accurately.


Recognition performance also improved for rare surgical actions that appear relatively infrequently in the training data. For actions that occur infrequently, such as applying a clip to prevent bile leakage before gallbladder removal, average precision rose from 33% to 39% on gallbladder surgery videos and from 43% to 56% on liver surgery videos.


The research team also created a new liver resection video dataset, "LLS48." It comprises approximately 52 hours of footage from 48 laparoscopic liver resections performed at Samsung Medical Center between February 2020 and March 2022. The videos were divided according to international standards for surgical video classification into five surgical stages, 18 subtasks, and 195 actions. Two specialists in hepatobiliary and pancreatic surgery and three experienced professionals with more than 10 years of experience took part in classifying and validating individual video scenes.


The team also confirmed that AI trained on detailed actions can identify the broader progression of surgery. It achieved an accuracy of more than 93% in predicting the surgical stages of gallbladder removal procedures. The team said this demonstrates the potential to learn the hierarchical structure of the surgical process, going beyond recognition of individual actions.


"For surgical robots to work naturally with medical staff, they must do more than identify surgical instruments or organs; they must understand what procedure is currently underway and what the medical staff are doing," said Jeong Gyuhwan, director of the center. "This study is significant because it presents AI technology that can recognize everything from individual surgical actions to the overall flow of an operation in an integrated way."


"In effect, AI that learned the movements of the fingertips has also learned to read the flow of an entire operation," Professor Oh Namgi said. "The technology could be used for automated surgical records, post-operative review, and resident training, and ultimately for humanoid surgical-assistance robots that recognize what is happening during an operation in real time."



The study was conducted with support from the Korea Health Industry Development Institute's Korea Advanced Research Projects Agency for Health (ARPA-H) project under the Ministry of Health and Welfare; a support program of the National Research Foundation of Korea under the Ministry of Science and ICT; Samsung Medical Center's Future Medicine 2030 Project; and the center's research funding. The findings were published in the international medical imaging journal IEEE Transactions on Medical Imaging.


This content was produced with the assistance of AI translation services.

© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.

Today’s Briefing