Millions of websites are set to gain new protections against AI bots, thanks to a “game-changing” move by internet infrastructure firm Cloudflare. The company, which hosts around 20% of the internet, is rolling out a new system that allows websites to block AI crawlers from accessing content without permission.
The system is already live on over a million sites, including major platforms like Sky News, Buzzfeed, and The Associated Press. It targets AI bots automated programs that scrape websites to collect data used to train AI systems. These bots, unlike search engine crawlers, often take content without providing any visibility or traffic back to the original site.
Cloudflare’s CEO Matthew Prince said the goal is to restore control to content creators and build a fairer online economy. “If the Internet is going to survive the age of AI, we need to give publishers the control they deserve,” he stated. The company is also developing a “Pay Per Crawl” model to let AI firms compensate publishers for access to their content.
This move comes amid growing backlash from creatives writers, artists, musicians who accuse AI companies of using their work without permission. The issue has sparked legal battles and political debates, including a notable dispute in the UK involving artists like Sir Elton John.
Publishers have long accepted search engine crawlers like Google’s, which help drive traffic to their sites. However, AI bots typically extract data without sending users to the source, depriving creators of both traffic and revenue.
Some experts, however, caution that tech solutions alone aren’t enough. Ed Newton-Rex, founder of Fairly Trained, likened Cloudflare’s system to “body armour that stops working when you leave your house,” calling instead for stronger legal protections.
Nevertheless, industry leaders like Condé Nast’s CEO Roger Lynch praised the move as a critical step in supporting quality journalism and creator rights.
With AI-generated traffic now exceeding 50 billion requests daily on Cloudflare’s network, the company’s new tool marks a major turning point in the ongoing battle between content creators and AI firms.
