S2W Paper on LLM Tokenizer Accepted at 'EMNLP 2025'

Insights Offered for the Discussion on "Sovereign AI"

S2W announced on the 10th that a paper co-authored by S2W and a research team from the Korea Advanced Institute of Science and Technology (KAIST), which identified a fundamental vulnerability in the tokenization structure of large language models (LLMs), was accepted at the Conference on Empirical Methods in Natural Language Processing (EMNLP) 2025.

EMNLP is considered one of the world's top three academic conferences in the field of natural language processing (NLP). S2W has achieved the distinction of having papers accepted at the world’s most prestigious artificial intelligence (AI) conferences for four consecutive years.

S2W announced on the 10th that a paper co-authored with a research team from the Korea Advanced Institute of Science and Technology (KAIST) was accepted at the Conference on Empirical Methods in Natural Language Processing (EMNLP) 2025.

The paper presented by the S2W research team, titled "Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers," reveals that the tokenizer-a tool that segments and processes characters during sentence analysis, and one of the processing functions inside LLMs-can induce hallucinations. The study specifically suggests that such hallucinations occur more frequently in languages other than English, raising the possibility that the quality of model responses could be further degraded in non-English contexts.

Park Keun-tae, Chief Technology Officer (CTO) of S2W, stated, "This paper provides meaningful insights into the ongoing discussion about 'sovereign AI,' which emphasizes the need for each country to develop and operate AI based on its own language and data. S2W plans to continue producing leading research results aimed at building trustworthy AI."

This content was produced with the assistance of AI translation services.