Motif Files Objection After Elimination in Second Round of 'National AI' Project: "Detailed Evaluation Scores Must Be Disclosed"

"Top AAII Benchmark Score, Yet Eliminated...
Difficult to Understand Evaluation Basis"

Motif Technologies, which was eliminated in the second phase assessment of the government's Independent Artificial Intelligence (AI) Foundation Model Project, has officially filed an objection, demanding that the evaluation criteria and detailed itemized scores be disclosed.

On the 27th, Motif issued a statement, saying, "With regard to the selection results for the second phase of the Independent AI Foundation Model Project, we have officially submitted an objection."

Motif requested the Ministry of Science and ICT to disclose the following: ▲the evaluation results and detailed evaluation contents, ▲the basis for the calculation method of the benchmark evaluation scoring, ▲the specific criteria and results of the expert evaluation, and ▲the methodology, detailed criteria, and results of the user evaluation.

Previously, the Motif 3 AI model, which Motif publicly released prior to the second evaluation, had received the highest score among the companies participating in the second-phase Independent Foundation Model assessment, according to benchmarks from a global evaluation agency. Specifically, the model achieved a score of 47 in the Artificial Analysis Intelligence Index (AAII), surpassing competitors such as SOLAR Open 2 (37 points, Upstage), A.DOT-X K2 (35 points, SK Telecom), and K-EXAONE 2.0 (31 points, LG AI Research).

The AAII was used as a key indicator for benchmark evaluation, accounting for 40 out of 100 possible points in the second-phase assessment. More specifically, AAII contributed 25 points, and the benchmark of the National Information Society Agency (NIA) contributed 15 points. Motif received a top converted score of 11.9 points on the AAII benchmark evaluation. However, in the overall ranking, the company received the lowest score and was eliminated.

Motif raised questions regarding the method of assigning benchmark scores. Despite the significant difference between Motif (47 points) and LG AI Research (31 points) in the AAII, the company emphasized that there is a need for a specific explanation about why and how a discrepancy occurred between the results of the global benchmark and those of the expert evaluation.

The company also demanded the disclosure of the grounds for expert evaluation decisions. Motif claims it was asked questions during the expert evaluation process, such as whether there is a company with foreign capital or an overseas corporation holding voting rights as a major shareholder among the lead or participating companies in the consortium. Motif argued that it is necessary to explain the purpose and criteria by which such questions were used in the evaluation.

Motif stated, "According to media reports, the Ministry of Science and ICT explained that Motif 3 only scored highly on the AAII benchmark, but had issues in the expert and user evaluations." The company continued, "A senior official at the ministry even commented that Motif 3's actual inference performance was lacking compared to its benchmark score, and that its usability and utility were found to be insufficient in evaluation."

They added, "This issue goes beyond simply distinguishing which company was selected and which was eliminated, and raises fundamental questions about the nature of the Foundation Model Project and the future direction it should pursue. We believe there may have been some misunderstandings about the technological direction we have pursued and its outcomes. With these concerns in mind, we respectfully request the detailed disclosure of evaluation scores and a re-review."

However, Motif clarified that this objection is not intended as an effort to participate in the upcoming third-phase evaluation of the Independent Foundation Model Project. The company stated, "Regardless of the outcome of our objection, we will not participate in the third phase. Our objection is not a claim that we must be selected. Rather, we hope this objection serves as a process for achieving consensus on the purpose of the Foundation Model Project and for our company and other competitors to agree on and find improvements going forward."

This content was produced with the assistance of AI translation services.