"Fake SNS Accounts Posing as Americans... Iran, China Take Influence Operations to New Level"

"Uncontrollable" Chinese Open-Source AI Models
Unlike Closed Models, Developers Cannot Exercise Control
Fueling Influence Operations... Cases Using DeepSeek Also Noted

It has been reported that Iran and China have combined Chinese open-source artificial intelligence (AI) models with AI agents to mass-produce fake accounts on social networking services (SNS) and use them to manipulate public opinion within the United States.

On September 18 (local time), The New York Times (NYT), citing U.S. government officials and security researchers, reported that Iran and China, as well as some Israeli private companies, recently deployed AI agents based on Chinese open-source AI models for online influence operations.

They created AI agents using Chinese open-source AI models and deployed hundreds of these agents on SNS platforms. The AI agents automatically created fake accounts on Instagram, Facebook, X (formerly Twitter), and TikTok, and wrote and spread false posts regarding political and social issues. The AI also coordinated the messaging across multiple accounts.

U.S. officials explained that this was virtually the first case in which AI agents were responsible for the entire influence operation process, from creating fake accounts and generating content to coordinating messaging. It was also reported that operators disabled most of the safety mechanisms intended to prevent AI from creating fake accounts or manipulating online opinion.

Although the operations themselves were still rudimentary, U.S. intelligence agencies and major tech companies are closely monitoring the situation. Investigations have found that operations by China and Iran received state support, while the two instances in Israel were led by private companies.

The U.S. government is also concerned that this method could be used to target American voters ahead of the upcoming midterm elections in November. While Russia, Iran, and China have previously carried out online influence operations during U.S. elections, recent advances in AI technology have enabled these activities to be massively automated with far fewer people.

Reuters Yonhap News

U.S. intelligence agencies are also paying close attention to the possibility of Iranian operations. Fake accounts created by Iranian AI agents posed as ordinary Americans living in major U.S. cities. These accounts tagged journalists and politicians while spreading popular memes and content criticizing the Republican Party. It is reported that in the first half of this year, the accounts engaged in such activities amassed about 80,000 followers. China has previously been identified as having tested similar AI tactics to those used by Iran.

There were also confirmed cases involving private Israeli firms. IntelEye, an AI security firm in Tel Aviv, used the open-source AI model from China’s DeepSeek to control thousands of SNS accounts. The company stated that it purchased 10,000 similar accounts and that AI agents actually operated about 1,000 of them.

The AI posted comments on posts and videos, attempted to converse with users, and even selected what topics to address autonomously. The accounts posted content supporting or opposing Israeli Prime Minister Benjamin Netanyahu, using the upcoming Israeli general election as a theme.

However, IntelEye claimed that the activity was an experiment to check AI safety rather than a true influence operation. Meta classified this activity as an "organized attempt to create fake accounts across multiple platforms" and stated that the majority of related accounts were removed by its automated detection systems.

Experts warned that the proliferation of Chinese open-source AI models could make such operations more difficult to control. Unlike closed models from companies like OpenAI or Anthropic, open-source models can run directly on personal servers and devices, making it hard for developers to block use or reapply safety mechanisms.

Kyle Crichton, a researcher at Georgetown University's Center for Security and Emerging Technology, stated, "AI agents can become highly sophisticated. Not only can they scale up, but they can also blend in to appear like real human users."

While there have been previous cases involving the use of AI to generate fake images, videos, and text, NYT pointed out that the rise of AI agents capable of not just creating but also operating fake accounts and running the entire influence campaign represents a new stage.

This content was produced with the assistance of AI translation services.