SKT Ranks Third, LG AI Research Fourth; Second Evaluation Results Unaffected
Controversy Over "Benchmarking" That Boosts Only Benchmark Scores

In the second evaluation round of the government-led Foundation Model for Independent Artificial Intelligence (known as Dokpamo), two participating startups outperformed two large corporations in a global performance benchmark, resulting in a surprising outcome.


According to Artificial Analysis, a global AI performance evaluation institute, on August 13, Motif Technologies’ large language model (LLM) “Motif3” received a score of 47 on the Artificial Analysis Intelligence Index (AAII), which is calculated by the institute. This was the highest score among the models submitted by the companies participating in the second Dokpamo evaluation. The AAII is one of the key metrics for the second round of evaluation.


Motif and Upstage Models Take Top Two Spots in Global Benchmark Among Dokpamo Participants View original image

The runner-up was Upstage. “Solar Open2,” the model Upstage submitted for the second evaluation, scored 37 points. Next was SK Telecom’s “A.DotX(A.X)-K2” with 35 points, followed by LG AI Research’s “K-EXAONE (EXAONE) 2.0” at 31 points. Notably, LG AI Research had taken first place in the first round of evaluations with a cumulative benchmark score of 33.6, but the model it submitted for the second round received the lowest score among domestic models in the Artificial Analysis evaluation.


However, the AAII scores in this round do not directly determine the final outcome of the second Dokpamo evaluation. The AAII is a composite index method, combining multiple separate benchmarks in areas such as knowledge, reasoning, and coding into a single score. Depending on which benchmarks are combined, there can be significant fluctuation in the result. Additionally, there has been controversy over so-called “benchmarking” — cases where only the benchmark scores are raised without reflecting the real model performance.


In the actual second Dokpamo evaluation, benchmark assessments account for 40 out of a possible 100 points. Of these, the AAII is responsible for 25 points while the Korea Information Society Agency (NIA) benchmark covers the remaining 15 points. The rest of the score is made up of expert evaluation (35 points) and user evaluation (25 points).


One limitation, particularly noteworthy, is that the AAII does not reflect Korean language proficiency, which is a key factor in the second Dokpamo evaluation. Indeed, language comprehension across multiple languages, including Korean, overall performance in specific industry tasks, or operational efficiency are difficult to estimate using only the AAII.



The government is expected to announce the results of the second Dokpamo evaluation soon, following the closure of the public review on August 12. In this round, one out of four participating teams will be eliminated, while three teams will survive and advance to the next stage.


This content was produced with the assistance of AI translation services.

© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.

Today’s Briefing