Robot AI Now Understands the Roles of Handles and Buttons [Reading Science]

GIST Develops 3D Recognition Technology Linking Objects, Components, and Functions
Component Search Accuracy Reaches 83.6%

A new artificial intelligence (AI) technology has been developed that enables robots not only to locate a microwave oven, but also to understand that the handle is used to open the door and the buttons serve to operate the device. This technology links the positions of objects with the functions of their components, allowing robots to translate human instructions into concrete tasks.

On September 17, the Gwangju Institute of Science and Technology (GIST) announced that the research team led by Professor Ewhan Kim of the Department of AI had developed the "Unified 3D Scene Graph (Unified 3DSG)," which integrates the locations, functions, and possible actions of objects and their components.

Actual Robot Utilization of Integrated 3DSG. The mobile robot "Stretch 3" recognizes the surrounding environment and connects human instructions to work plans and movement targets based on the functions of objects. Provided by the research team.

The 3D scene graph is a kind of 3D knowledge map that connects the positions and interrelationships of objects, enabling robots to understand their surroundings. Existing technology focused on recognizing entire objects, such as chairs or microwaves, but had limitations in identifying the positions and functions of smaller components like handles and buttons.

The "OP3DSG" developed by the research team first detects objects in camera images, then searches for the detailed components that make up those objects. For example, if it finds an oven, it will then additionally identify features such as the door, handle, and buttons.

Videos taken from multiple angles are integrated into a three-dimensional space. During this process, the locations, meanings, and colors of components are compared to match identical parts that appear in different images. Afterward, a large language model (LLM) is used to supplement the functional relationships and possible actions between objects and components.

For instance, to spatial information that connects a microwave and a handle, functional data—such as the fact that a handle is used to open the door—can be added.

Component Search Performance: 83.6%... Applied to Real Robots

When tested with the team’s in-house evaluation system "UniGraph3D," the correct component was included among the top three candidates suggested by AI in 83.6% of cases. This is 31.2 percentage points higher than the previous best method, which yielded a top score of 52.4%. Performance in identifying the spatial relationships between objects and components improved by 7.9 percentage points, and understanding functional relationships improved by 9.2 percentage points.

The technology was also applied to the actual mobile robot "Stretch 3." The robot successfully identified its surroundings and was able to move following human instructions.

The robot could, for example, count the number of chairs in its environment or locate a remote control needed to turn on the TV and navigate to its position. When instructed to "tidy up the laundry," it planned a series of actions that included finding towels and putting them in a basket.

Research team photo. (From left) Ewhan Kim, Professor of AI Department at GIST; Irum Kim, Integrated Master’s and Doctoral Program Student. Provided by GIST

When given the instruction, "I am thirsty. Please bring me something I can use to drink," the robot used the functional relationships of objects to determine the necessary item and set a movement target accordingly. However, it should be noted that this demonstration focused on object detection, navigation, and work planning, and did not verify the robot's ability to physically manipulate every object in reality.

Ewhan Kim, Professor of the GIST AI Department, commented, "The core of this research is that the robot integrates not only where things are, but also which components can be used, and how they can be used, all into a single source of information. We expect this foundational technology will, in the future, allow service robots to understand complex real-world environments and turn human natural language instructions into actionable work plans and navigation."

This research, with Irum Kim, an integrated Master’s and Doctoral program student at the GIST AI Department, as the first author, was presented at the international computer vision conference "ECCV 2026," which took place in Malmö, Sweden, from August 8 to 12.

This content was produced with the assistance of AI translation services.