"Don't Use My Content to Train AI"... Google, OpenAI Allow Blanket Blocking

Standard introduced to opt out of data collection for AI training
Block data collection for AI training while maintaining search visibility

Website operators will soon be able to block the use of their content for artificial intelligence (AI) training with a single setting. Major AI companies, including Google and OpenAI, have agreed to adopt a common standard that lets website operators opt out of having their materials collected for training.

OpenAI, Google and Microsoft (MS) said on Nov. 1 that they would support a common standard allowing website operators to control the collection of materials for AI training in one place.

Previously, website operators had to block each AI company’s crawler—a program that automatically collects information from websites—individually through a “robots.txt” file. They also had to add new entries to their blocklists whenever a new bot appeared.

Under the new standard, a single setting to reject the collection of materials for AI training will block crawlers from major AI companies. It will also be possible to restrict the collection of materials for AI training while keeping content visible to search engines.

In an official blog post, Microsoft said, “Website operators should not be forced to choose between making their content available to search engines and deciding how generative AI uses their work.” OpenAI also described the move as “a practical step toward an open web.”

The standard was developed based on discussions at the Internet Engineering Task Force (IETF). Anthropic has already implemented it, while Google and Microsoft plan to adopt it next year, and OpenAI plans to do so within the coming months. Meta said it supported the initiative’s goals but had not decided whether to participate.

The move comes as media companies and publishers have filed a series of lawsuits in protest against AI companies’ unauthorized collection of content.

This content was produced with the assistance of AI translation services.