Gender Equality Ministry: "AI Character Chatbots Found to Encourage Crime and Illegal Acts... Youth Protection to Be Strengthened"

A review of conversations generated by AI character chatbots found
Specific depictions of sexual acts, violence and brutality
Stronger cooperation with operators to create a safer online environment

The Ministry of Gender Equality and Family said on Oct. 15 that some responses generated by AI-powered character chatbots that converse with adolescents contained harmful information, including sexual and violent depictions, and content encouraging self-harm, suicide, and crime. The ministry has asked operators of AI character chatbot services to voluntarily improve their measures to protect young people.

AI chatbot file photo. The image is unrelated to the article.

AI character chatbots are services in which users select or create characters with different personalities and story settings, then continue conversations and stories through responses generated by AI.

Some AI character chatbot services have age-verification features, but users can also talk with characters in general-use settings without verifying their age. This makes it necessary to assess whether the responses actually generated are harmful to young people.

The Center for Protection of Youth Media Environment, operated under contract by the Korea Youth Counseling & Welfare Institute on behalf of the Ministry of Gender Equality and Family, examined conversations generated by characters on some AI character chatbot services.

After reviewing the responses against the criteria for deliberating on media products harmful to young people under Article 9 of the Juvenile Protection Act and Appendix 2 of its Enforcement Decree, the center found that some characters generated responses that could be harmful to young people, including sexual and violent depictions, and content encouraging self-harm, suicide, and crime.

The main types of concerning responses identified included ▲specific depictions of sexual content and sexual acts ▲violence and brutality ▲content encouraging self-harm, suicide, crime, and illegal acts. Some characters generated responses raising concerns across multiple categories.

The Ministry of Gender Equality and Family shared the findings with the relevant operators and asked them to voluntarily improve measures to protect young people.

In particular, it called on operators to adjust the scope of search, recommendations, and conversations available for characters that may generate inappropriate dialogue for young people, according to whether users have verified their age. It also asked operators to strengthen safeguards so that content posing a serious risk of harm, such as responses encouraging self-harm or suicide or helping users carry out specific crimes, is not generated regardless of the user's age.

The ministry plans to continue working-level consultations with operators, regularly check their self-assessments and improvements to protective measures, and strengthen cooperation to ensure that safeguards for young people work effectively in practice.

A ministry official explained, "If content is designated as harmful to young people under the Juvenile Protection Act, the Broadcasting Media Communications Review Commission can request that it be deleted or blocked. However, content generated through conversations between users and AI chatbots is not made public, so it is difficult to request the removal of individual items. That is why we asked operators to regulate their services voluntarily." The official added, "If voluntary improvements are not made, we also plan to take follow-up measures, such as requesting removal through the Broadcasting Media Communications Review Commission."

Yoon Sejin, the ministry's Director General for Youth Policy, said, "Operators' active efforts to improve their services and ongoing oversight are important to protect young people using AI character chatbot services. We will strengthen cooperation with operators and work to create an online environment where young people can use these services more safely."

This content was produced with the assistance of AI translation services.