Traditionally, websites have relied on the honor system using a file called robots.txt to grant or restrict access to scrapers. However, various AI companies, including prominent ones like Anthropic and Perplexity AI, have been reported for disregarding these directives. Cloudflare disclosed that it deals with more than 50 billion web crawler requests daily and, while it possesses tools to pinpoint and block malicious activities, this leads perpetrators to adapt their tactics continuously in an unending battle.

Rather than simply blocking bots, Cloudflare’s AI Labyrinth forces them to engage with irrelevant data, distinct from the actual content of the website. Described as a “next-generation honeypot,” this tool attracts AI crawlers into navigating towards fictitious pages in-depth, a behavior unlikely for genuine human users. This innovative approach aids in identifying and categorizing malicious bots for Cloudflare’s blacklist while uncovering new bot behaviors and markers that might otherwise go unnoticed. These misleading links are crafted to remain invisible to regular human visitors, serving as an effective strategy against unauthorized web scraping.

You can read more about how AI Labyrinth works on Cloudflare’s blog, but here’s a bit more detail from the post:

We found that generating a diverse set of topics first, then creating content for each topic, produced more varied and convincing results. It is important to us that we don’t generate inaccurate content that contributes to the spread of misinformation on the Internet, so the content we generate is real and related to scientific facts, just not relevant or proprietary to the site being crawled.

Website administrators can opt into using AI Labyrinth by navigating to the Bot Management section of their site’s Cloudflare dashboard’s settings and toggling it on. The company says that this “is only the first iteration of using generative AI to thwart bots.” It plans to create “whole networks of linked URLs” that bots that end up in will have a hard time clocking as fake. As Ars Technica notes, AI Labyrinth sounds similar to Nepenthes, a tool that’s designed to sideline crawlers for “months” in a hell of AI-generated junk data.