Cloudflare Blocks AI Crawlers & Bots By Default - LLMrefs

If your website uses Cloudflare and you have not explicitly allowed AI crawlers, your site is invisible to AI search engines like ChatGPT, Perplexity, and Claude.

On July 1st 2025, Cloudflare flipped a switch that changed how 20% of the public web interacts with AI systems. Every new Cloudflare domain now blocks all known AI crawlers by default.

This is not a small change. Cloudflare protects roughly one fifth of all websites on the internet. That means a significant portion of the web just moved from "AI can scrape by default" to "AI needs explicit permission".

Why Did Cloudflare Block AI Bots?

AI crawlers take a lot and give almost nothing back. Website owners were watching their content get used to train AI models and power AI search results. They received almost no traffic or compensation for it.

Cloudflare published data showing just how bad it has gotten. Anthropic's ClaudeBot makes approximately 71,000 requests for every single referral click it sends back to websites.

Cloudflare is calling this change "Content Independence Day." Website owners should control how AI systems use their content. They should not have to opt out of scraping that happens by default.

What This Means For AI Search Visibility

ChatGPT and other AI answer engines are becoming major traffic sources. If you block AI crawlers completely, you become invisible to this channel. Track your AI search visibility to see how blocking or allowing crawlers affects your brand's presence in AI-generated answers.

AI search tools like ChatGPT, Perplexity, Claude and Microsoft Copilot need to crawl your content before they can cite you. When Cloudflare blocks these crawlers, your content disappears from AI-generated answers.

If you want visibility in AI search, this default blocking is a problem. Your competitors who allow AI crawlers will be the sources these systems use instead.

Reasons to Block AI Crawlers

Protecting content value. If your business model depends on users visiting your site to consume content, AI summaries may cannibalize that traffic. Users get their answer from ChatGPT without ever clicking through.
Preventing unauthorized training. Your content might be used to train competing AI models. Blocking prevents this use while you evaluate your options.
Future monetization. As pay-per-crawl models mature, maintaining blocking gives you leverage to negotiate compensation.

Reasons to Allow AI Crawlers

AI search visibility. If ChatGPT cannot see your content, it cannot cite you. For brands seeking visibility in AI-generated answers, blocking is counterproductive.
Competitive advantage. While competitors block access, your content becomes the source AI systems draw from. Smaller sites can gain influence by remaining accessible.
Traffic opportunity. AI search referrals are growing. Blocking eliminates this traffic source entirely.

Should You Block AI Bots On Your Site?

It depends on what you want from your website. Some sites will benefit from blocking AI bots, whilst others will benefit from allowing AI bots.

When to block AI crawlers	When to allow AI crawlers
Keep users on your site	Get cited in AI answers
Control how your content is used	Become the go-to source
Negotiate payment for access	Capture AI referral traffic

AI Crawler Settings Available In Cloudflare

Cloudflare gives you several ways to control AI crawler access. You can block them entirely, allow specific ones, or even charge for access.

Block AI bots

The most straightforward option is the Block AI Bots feature. This setting prevents verified AI crawlers and unverified bots with similar behaviour from accessing your site.

You have three configuration options.

Do not block (off). AI bots can access your entire site without restrictions
Block on all pages. AI bots are blocked across your entire site
Only block on hostnames with ads. AI bots are blocked only on pages displaying advertisements

The "Block on all pages" setting is now the default for newly created Cloudflare domains.

AI crawl control

AI Crawl Control gives you visibility into which AI services are accessing your content. You can see which crawlers are visiting, how often they make requests, and whether they respect your robots.txt directives.

This tool lets you make granular decisions. You might choose to allow GPTBot for search visibility while blocking training-focused crawlers. Or you might allow all AI access while monitoring which services actually send referral traffic.

The dashboard shows you which crawlers violate your robots.txt file. This information helps you understand which AI companies respect your preferences and which ignore them.

AI labyrinth

Some AI crawlers ignore robots.txt entirely. They scrape whatever they want regardless of your stated preferences.

AI Labyrinth is Cloudflare's answer to this problem. The feature adds invisible links to your pages that create a maze for unauthorized crawlers. Human visitors cannot see these links. Compliant bots following robots.txt ignore them. But scrapers that disregard your directives get trapped in an endless loop of fake pages.

The links use nofollow tags so they do not affect your SEO. The feature is designed to waste the resources of bad actors without impacting legitimate visitors or search engines.

Pay per crawl

Cloudflare has proposed a marketplace model where AI companies pay website owners for crawler access. The concept is simple. If an AI company wants to scrape your content for training data or to power their products, they should compensate you.

When you enable this feature, blocked crawlers receive a 402 Payment Required response. This signals to AI operators that your content is available for a price. Cloudflare's marketplace would then handle the transaction between website owners and AI companies.

This feature is still new. Pricing and participation are evolving. But it points to a future where content creators get paid for the value they provide to AI systems.

Content signals policy

Cloudflare has updated the traditional robots.txt system with more specific controls for AI use cases. You can now set explicit permissions for three distinct activities.

ai-train. Whether your content can be used to train AI models
search. Whether your content can appear in search results
ai-input. Whether your content can be fed into AI systems for inference

These controls matter because the use cases are different. You might want your content in ChatGPT search results but not in training datasets. These signals let you set that preference.

How To Allow AI Crawlers In Cloudflare

If your site uses Cloudflare, you need to change two settings to enable AI crawlers to access your content.

AI training bots
AI bot traffic

The steps differ slightly depending on whether you are adding a new domain or configuring an existing one.

For new domains

When you add a new domain to Cloudflare, you will see the AI crawler settings during the setup flow. Look for the "Control how AI crawlers access your site" section.

Select "Do not block (off)" under Block AI training bots
Turn off the "Instruct AI bot traffic with robots.txt" toggle
Click Continue to complete your domain setup

For existing domains

If your domain is already on Cloudflare, you need to find the Control AI crawlers settings in your dashboard.

Log in to the Cloudflare dashboard
Select your account and domain
Find the Control AI crawlers section
Set "Block AI training bots" to "Do not block (allow crawlers)"
Set "Manage your robots.txt" to "Disable robots.txt configuration"

These settings ensure AI crawlers like GPTBot, ClaudeBot, and others can access your content for AI search visibility. You can check if AI bots can read your pages with our free tool.

Conclusion

The era of unlimited free scraping is ending. Cloudflare's decision signals where the industry is heading.

If you want AI search visibility, opt in now. If you want to protect your content, keep the defaults. Either way, check your Cloudflare settings and make an active choice rather than accepting whatever was configured by default.