[Park Jongseong's AI Frontier] Perfect Score on Exams, 12 in Practice... Why AI Lost Its Way in a Pile of Documents
Real-World Capabilities Hidden Behind a Glittering Report Card
Before Cutting Staff, First Identify Tasks for Human-AI Collaboration
Companies That Manage People Well Also Make the Best Use of AI
Since it first emerged, artificial intelligence (AI) has consistently had a glittering report card.
In July 2025, models from Google DeepMind and OpenAI scored at the gold-medal level at the International Mathematical Olympiad. Two months later, an evaluation released by OpenAI, called "GDPval," concluded that the models "performed as well as or better than experts on nearly half of real-world job tasks." GDPval measures AI's ability to perform representative tasks from 44 occupations across nine industries that make significant contributions to U.S. gross domestic product (GDP).
But the report card looked quite different in a new evaluation, "GDPx," released on Nov. 2, 2026, by SemiAnalysis, a U.S.-based semiconductor and AI analysis firm. The top models touted by Anthropic, Google and OpenAI took part. Each model could attempt each task three times. A task earned 100 points if completed and zero if not. There were 11 subjects in total. Scores were averaged within each subject, and those averages were then averaged again to calculate the final score. The results: Anthropic's model scored 12, Google's 10.9 and OpenAI's 9.6. These scores fell far short of the maximum average score of 100.
Why was there such a difference? According to SemiAnalysis, GDPx was designed to focus more on carrying out practical tasks than on theory. In the evaluation, AI models were sent into complex workplace scenarios where they had to find the necessary clues themselves. Working with 14 specialist startups, the organizers created 206 tasks across 11 subjects, including semiconductor design, biology, finance, law, and quantitative trading, which uses mathematical models to invest. They also built work environments for each task that closely resembled real workplaces. In finance, one task was based on an actual transaction: a private equity firm was to acquire about 62% of a family-owned bakery, with additional payment tied to the company's performance. Multiple models attempted the task 25 times, but none completed it. In 15 attempts, they incorrectly classified the success fee payable to the investment bank upon closing the deal as a liability the company had to repay, despite the acquisition agreement stating, "Treat this fee separately as a transaction cost, not as debt." In 17 attempts, they treated salaries paid to family members who had never worked a single hour as company operating expenses.
Exam questions contain all the information needed to solve them and have a single correct answer. Real-world work is different. You have to decide for yourself which documents matter among thousands, and anticipate what moves the other party to a transaction might make. This is precisely where the evaluation exposed AI's weakness. In particular, AI failed by choosing which information to trust based on which document stated the numbers most clearly, rather than on which document was authoritative. It trusted an old spreadsheet prepared by an adviser more than a contract signed by both parties. An experienced accountant, by contrast, would spot a suspicious line on a payroll statement and first check which document was legally binding. AI still lacks that kind of discernment.
Of course, the evaluation scores cannot be taken as a complete reflection of AI's capabilities. AI also has clear strengths and demonstrated remarkable problem-solving ability on some tasks, including math. In the end, the issue comes down to how it is used. OpenAI founding member Andrej Karpathy called AI "jagged intelligence," meaning it has clear strengths and weaknesses. This is a reminder that AI should be used selectively, rather than trusted wholesale or discarded. It is a lesson companies should take to heart.
It is still too soon to conclude that, because AI is already doing many tasks and will take on even more, companies should hire fewer people or lay them off. The first step should be to determine which tasks are suitable for AI. Companies need to distinguish between tasks that should be assigned to AI, those that people should do, and those people and AI should do together. Then they should expand and redesign roles around the work that people need to do. Their talent development plans should also reflect this. AI can take on routine tasks, freeing up people's time. Companies would benefit from using that time to help employees improve their work performance and to develop or recruit people who can exercise the discernment AI lacks. Ultimately, companies that manage people well will also make the best use of AI. They should keep this in mind.
Hot Picks Today
"Two Spend 500,000 Won in a Flash, Yet Wait Two Hours?" ... 'Dreamland' Shunned by Kids and Foreign Tourists
- The Era of Getting 200 Million Won After 30 Years Is Over... A Japanese Bank's "Perpetual Department Head" Who Just Clocked In Cries Out
- Spain Hit by Its Worst Tragedy... Lee Kangin, Moreno and Real Madrid Join Soccer World in Mourning
- They Said, “Make It Your Home for Life” - Then Suddenly Kicked 200 Passengers Off... What Happened to the Around-the-World Cruise?
- "130 Rice Bowls Sold Out in a Day"... 'Unfiltered Face' Adds 8 Million Won to Night Market Sales: China's Bungeunwoo
Park Jongseong, Writer and AI Consulting Leader
© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.