The new Pay Per Crawl initiative allows publishers to set fees for AI companies to access their content. Interested AI firms can view these fees and decide if they want to pay to access the content or not. Currently, this service is limited to a selected group of prominent publishers and creators, aimed at ensuring AI companies access superior content legitimately and fairly.

Cloudflare has been assisting domain owners combat AI crawlers for some time. While the option to block AI crawlers was introduced in 2023, it initially only applied to those respecting a site’s robots.txt file, a non-mandatory protocol indicating bot access preferences. Last year, Cloudflare expanded its services to block all AI bots, regardless of the robots.txt file obedience, a default setting for new customers. The company distinguishes scrapers using a registry of known AI bots and introduced a feature in March where web-crawling bots are misdirected through an “AI Labyrinth” to prevent unauthorized scraping.

Several major publishers and online platforms, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, are on board with Cloudflare’s new AI crawler restrictions, as websites contend with a future where more people are finding information through AI chatbots, rather than search engines. “People trust the AI more over the last six months, which means they’re not reading original content,” Cloudflare CEO Matthew Prince said during the Axios Live event last week.

Additionally, Cloudflare says it’s working with AI companies to help verify their crawlers and allow them to “clearly state their purpose,” such as whether they’re using the content for training, inference, or search. Website owners can then review this information and determine which crawlers to let in.

“Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it,” Prince said in the press release. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”