Kakao's Lightweight AI Model 'Kanana-2' Outperforms Google and Alibaba in Safety Evaluation

Outperformed Google Gemma and Alibaba Qwen in Comprehensive Safety Evaluation

Pre-release Safety Assessments to Be Applied to Future In-house AI Models

Kakao announced on August 18 that the safety evaluation results of the lightweight language model (SLM) Kanana-2 series, which was released as open source, outperformed major global models.

Kakao used its self-developed "AI Safety Evaluation Platform" to assess two models released as open source on Hugging Face on July 28: "Kanana-2-1.3B-Instruct" and "Kanana-2-3B-Instruct." The company measured factors such as harmfulness and bias. As a result, both models ranked first overall compared to major global models with similar parameter sizes.

Kakao announced on the 18th that the safety evaluation results of the lightweight language model (SLM) Kanana-2 series, released as open source, showed that it outperforms major global models. Kakao

For this evaluation, the Korean language-specific safety benchmark "AssurAI," jointly developed last November by the Telecommunications Technology Association (TTA), Korea Advanced Institute of Science and Technology (KAIST), and Kakao as part of a Ministry of Science and ICT project, was utilized. AssurAI is designed to fit the social and cultural context of Korea and can measure AI risk factors across various conditions, including text, image, audio, multimodal content, and malicious prompt scenarios.

AssurAI includes 9,560 evaluation items across a total of 35 risk categories. Kakao reorganized these into five broad classifications for analysis: social risk, sexual content and child protection, crime and illegal activities, violence, and infringement of rights. Additionally, the "LLM-as-a-Judge" approach, in which AI models themselves evaluate responses according to predefined scoring standards, was applied to enable model comparisons under consistent criteria even at scale.

For this evaluation, Kakao set global models of similar parameter size as comparison targets, specifically Google's Gemma and Alibaba's Qwen. The Kanana-2-1.3B-Instruct model achieved an overall score of 0.70, surpassing Gemma (0.68) and Qwen (0.58). Kanana-2-3B-Instruct also scored 0.70, higher than Gemma (0.66) and Qwen (0.62).

Kakao plans to use this evaluation as a starting point to further expand routine pre-release safety assessments for all in-house developed AI models to be released to the public in the future. The company also aims to broaden the scope of evaluations to cover not only text-based language models but also multimodal models and agentic AI risk assessment areas.

Kim Kyunghoon, Kakao AI Safety Leader, stated, "This evaluation was an effort to check the safety of open-source models in advance through our in-house verification system and to release those results. In the future, we will continue to establish safety verification as a core step in the AI model release process in line with responsible AI development principles."

This content was produced with the assistance of AI translation services.