Motif and Upstage Models Take Top Two Spots in Global Benchmark Among Dokpamo Participants
SKT Ranks Third, LG AI Research Fourth; Second Evaluation Results Unaffected
Controversy Over "Benchmarking" That Boosts Only Benchmark Scores
In the second evaluation round of the government-led Foundation Model for Independent Artificial Intelligence (known as Dokpamo), two participating startups outperformed two large corporations in a global performance benchmark, resulting in a surprising outcome.
According to Artificial Analysis, a global AI performance evaluation institute, on August 13, Motif Technologies’ large language model (LLM) “Motif3” received a score of 47 on the Artificial Analysis Intelligence Index (AAII), which is calculated by the institute. This was the highest score among the models submitted by the companies participating in the second Dokpamo evaluation. The AAII is one of the key metrics for the second round of evaluation.
The runner-up was Upstage. “Solar Open2,” the model Upstage submitted for the second evaluation, scored 37 points. Next was SK Telecom’s “A.DotX(A.X)-K2” with 35 points, followed by LG AI Research’s “K-EXAONE (EXAONE) 2.0” at 31 points. Notably, LG AI Research had taken first place in the first round of evaluations with a cumulative benchmark score of 33.6, but the model it submitted for the second round received the lowest score among domestic models in the Artificial Analysis evaluation.
However, the AAII scores in this round do not directly determine the final outcome of the second Dokpamo evaluation. The AAII is a composite index method, combining multiple separate benchmarks in areas such as knowledge, reasoning, and coding into a single score. Depending on which benchmarks are combined, there can be significant fluctuation in the result. Additionally, there has been controversy over so-called “benchmarking” — cases where only the benchmark scores are raised without reflecting the real model performance.
In the actual second Dokpamo evaluation, benchmark assessments account for 40 out of a possible 100 points. Of these, the AAII is responsible for 25 points while the Korea Information Society Agency (NIA) benchmark covers the remaining 15 points. The rest of the score is made up of expert evaluation (35 points) and user evaluation (25 points).
One limitation, particularly noteworthy, is that the AAII does not reflect Korean language proficiency, which is a key factor in the second Dokpamo evaluation. Indeed, language comprehension across multiple languages, including Korean, overall performance in specific industry tasks, or operational efficiency are difficult to estimate using only the AAII.
Hot Picks Today
"Two Spend 500,000 Won in a Flash, Yet Wait Two Hours?" ... 'Dreamland' Shunned by Kids and Foreign Tourists
- The Era of Getting 200 Million Won After 30 Years Is Over... A Japanese Bank's "Perpetual Department Head" Who Just Clocked In Cries Out
- Spain Hit by Its Worst Tragedy... Lee Kangin, Moreno and Real Madrid Join Soccer World in Mourning
- They Said, “Make It Your Home for Life” - Then Suddenly Kicked 200 Passengers Off... What Happened to the Around-the-World Cruise?
- "130 Rice Bowls Sold Out in a Day"... 'Unfiltered Face' Adds 8 Million Won to Night Market Sales: China's Bungeunwoo
The government is expected to announce the results of the second Dokpamo evaluation soon, following the closure of the public review on August 12. In this round, one out of four participating teams will be eliminated, while three teams will survive and advance to the next stage.
© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.