It Learned Language, Now It Must Learn the World... Physical AI's Next Battle [Science Scope]
Competition in VLA, World Models, and WAM... The Evolution of Robot Intelligence
KAIST Professor Im Seonghun: "Lack of Behavioral Data Is the Biggest Bottleneck"... An Opportunity for Korean Manufacturing
A robot powered by Google DeepMind's "Gemini Robotics 1.5" sorts clothes by color. Courtesy of Google DeepMind.
View original image"Pick up the cup and put it on the shelf."
It is a simple instruction for a person, but a difficult task for a robot. The robot must locate the cup and control the movement and force of its hand. It must also change its behavior if an obstacle appears. Understanding the meaning of words and acting on them in the real world are entirely different challenges.
Artificial intelligence (AI) is beginning to learn about the physical world, going beyond language. Physical AI, which perceives and moves through its surroundings, is emerging as a central focus of the next-generation technology race, moving beyond generative AI that writes and answers questions. But robots still have a long way to go before they can act like people. The biggest hurdle is their lack of experience accumulated in the real world.
In an interview with The Asia Business Daily, Im Seonghun, a professor at the Korea Advanced Institute of Science and Technology (KAIST) Graduate School of Culture Technology and founder of Neural Dynamics, said, "While language models learn patterns in language and knowledge from vast amounts of text, robots need to learn what results follow from taking certain actions in particular situations."
Turning Words into Actions... The VLA Challenge
Large language models (LLMs) such as ChatGPT have advanced by learning from vast amounts of text accumulated on the internet. Robots, by contrast, cannot learn how to pick up an object just by reading. They need to experience the object's weight and friction, as well as changes in the environment as it moves.
The technology driving this shift is the Vision-Language-Action (VLA) model. It surveys its surroundings through camera footage, understands a person's instructions, and then generates the robot's movements. This is a leading approach to robot foundation models trained for use across a range of tasks.
Lim Seonghun, professor at the Graduate School of Culture Technology at the Korea Advanced Institute of Science and Technology (KAIST) and founder of Neural Dynamics. Photo courtesy of KAIST.
View original imageWhile conventional industrial robots repeat predetermined movements, robots using VLA are designed to adapt when an object's location or work instructions change. Google's RT series of models and the open-source model OpenVLA are examples.
But even a robot that deftly picks up a cup in a laboratory may fail if the object's shape or lighting changes. The problem is even more complex in industrial settings, where materials, weight, and the surrounding environment can change constantly.
Professor Im said, "VLA has shown the potential to perform a variety of tasks with a single model and adapt to new environments." However, he noted, "We have yet to achieve a general-purpose model that can reliably handle exceptional situations in real industrial settings."
Predicting the Future Before Acting... The Emergence of World Models
Technology that enables robots to anticipate the results of their actions is also advancing. This is known as a "World Model." Just as a person can predict which direction a cup will move when pushed, a world model calculates how the surrounding environment will change when a specific action is taken in a given situation.
World foundation models such as NVIDIA's Cosmos are being developed to learn changes in physical environments and generate future scenarios. They can also be used to test situations that would be difficult to experience in real life in a virtual environment.
Recently, researchers have also been exploring "World Action Models (WAMs)," which combine future prediction with action generation. While VLA observes its surroundings and generates actions, a world model predicts how the world will change after an action. WAMs seek to predict both future situations and actions.
However, generating a plausible future scene does not necessarily mean accurately predicting how real objects will move. If a robot misjudges friction, contact, or shifts in the center of gravity, its actions may fail.
Fei-Fei Li, a professor at Stanford University and co-founder of World Labs, has also drawn attention to the potential of world models. In an interview with Bloomberg last September, Professor Li said, "There is still a major bottleneck in creating environments suitable for training and evaluating robots. Sophisticated world models could help address this bottleneck."
She added, "There isn't enough data in highly specialized fields such as contract manufacturing and laboratory research," explaining that virtual environments that closely resemble reality could be created using world models and used to train and evaluate robots.
Li Fei-Fei, a professor of computer science at Stanford University in the United States, leads research into "spatial intelligence," which enables AI to understand and predict the physical world. She is also co-founder and CEO of AI company World Labs. Courtesy of Stanford University.
View original imageSuccess in the Virtual World, Failure in Reality... The Robot's Dilemma
The fundamental problem facing Physical AI is data. Language models can learn from vast amounts of text accumulated on the internet, but data linking robot actions to their outcomes is relatively scarce and costly to obtain.
One common method is "teleoperation," in which a person remotely controls a robot. As the operator moves the robot, camera footage and joint movements are recorded. The robot then uses this data to imitate human actions through imitation learning.
However, data collection requires robots and personnel, as well as considerable time and money. It is also difficult to share data because robots differ in their joint configurations and body structures. Professor Im noted, "If the experience is limited or biased toward specific situations, there are limits to how much the model's ability to generalize can be improved."
This is why researchers are focusing on simulation. They can operate numerous robots in virtual spaces, vary the conditions, and train the robots. But repeatedly succeeding in a virtual world does not guarantee success in reality. This is because it is difficult to perfectly reproduce friction, contact, material deformation, and sensor errors. This is known as the "simulation-to-reality gap."
Professor Im explained, "Simulation can help make up for a lack of data, but it must be accompanied by experience and testing in real environments. The process needs to continue, with robots gaining experience in virtual environments and then being tested on real robots."
Learning Skilled Workers' Judgment... An Opportunity for Korean Manufacturing
Research is also expanding into ways of collecting behavioral data from videos of people working or wearable devices, without directly operating robots. In this approach, robots first learn work processes from first-person footage of people, then combine that information with a relatively small amount of robot data.
However, because human hands and robot hands have different structures, it is difficult to reproduce actions shown in videos exactly. That is why not only the quantity but also the diversity and quality of data matter. Robots must also learn how people recover from failed tasks and what judgments lead skilled workers to intervene.
A worker welds inside a confined structure at a shipyard. Physical AI technologies are being developed to enable robots to perform tasks in such narrow, hazardous spaces by transferring the judgment of skilled workers to them. Neural Dynamics website
View original imageProfessor Im believes opportunities can be found in Korea's manufacturing, shipbuilding, and construction industries. He said that turning skilled workers' experience and judgment into data that robots can learn from could give Korea a competitive edge.
The Physical AI startup Neural Dynamics, which he recently founded, also grew out of this perspective. The company is developing technology to convey skilled workers' judgment to robots in narrow, hazardous spaces at shipyards and accumulate it as training data.
The company's concept is to have a skilled worker mark welding positions in footage of a worksite captured by a robot. AI then converts the markings into a three-dimensional work path, which the robot executes after experts review, revise, and approve it. The goal is to record which paths were approved and revised and use that information for subsequent training.
However, the video released by the company is a concept demonstration created in a virtual simulation environment. It does not mean that the technology has been validated at an actual shipyard or commercialized.
From a Nation That Builds Robots to One That Teaches Them
Professor Im advised that, rather than aiming for general-purpose robots capable of performing every task from the outset, development should begin with tasks of industrial value, such as transporting parts or handling materials.
The key is a cycle in which robots are deployed at worksites, work data is collected, models are improved, and the robots are put back to work. This is similar to Tesla's approach of collecting data from vehicles driven on real roads to improve autonomous driving technology. However, making use of robot data is more complex because tasks and robot body configurations vary widely.
Other challenges include determining how much information about companies' production processes should be shared, how to integrate data from different robots, and how to verify the safety of learned behaviors.
Professor Im identified the continued collection of data for robots to learn from as a key challenge over the next 3 to 5 years. "Korea's competitiveness in Physical AI will depend on establishing a system that uses models in the field, collects data, improves the models, and then puts them back into use," he stressed.
Hot Picks Today
"Two Spend 500,000 Won in a Flash, Yet Wait Two Hours?" ... 'Dreamland' Shunned by Kids and Foreign Tourists
- The Era of Getting 200 Million Won After 30 Years Is Over... A Japanese Bank's "Perpetual Department Head" Who Just Clocked In Cries Out
- Spain Hit by Its Worst Tragedy... Lee Kangin, Moreno and Real Madrid Join Soccer World in Mourning
- They Said, “Make It Your Home for Life” - Then Suddenly Kicked 200 Passengers Off... What Happened to the Around-the-World Cruise?
- "130 Rice Bowls Sold Out in a Day"... 'Unfiltered Face' Adds 8 Million Won to Night Market Sales: China's Bungeunwoo
AI has already made remarkable progress in understanding what people say. The remaining challenge is to learn the complexities and exceptional situations of the real world. Ultimately, the next battle in Physical AI will be about how much varied experience robots can accumulate and how effectively they can turn failures into learning.
© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.