This guide comes from a real incident: we found this block active on the three sites we operate, with nobody having turned it on on purpose. The full incident is documented in the Servando case.
The symptom nobody understands
Your site ranks on Google. It loads fast. There are no errors in your logs. And yet, when someone asks ChatGPT or Claude to read your site, the AI replies that it cannot access it. Or worse: you never get cited and you don’t know why.
The cause is usually in a place nobody checks: two security settings that Cloudflare enables by default on every zone.
The cause: two settings that come switched on out of the box
Browser Integrity Check (BIC) challenges visitors whose browser “doesn’t look legitimate” according to header and behavior heuristics. A human browser answers the challenge without anyone noticing. An AI bot does not run JavaScript or store cookies: it gets a challenge page and leaves empty-handed.
JS Detection (part of the anti-bot modes) adds another layer of the same kind: a JavaScript challenge that AI crawlers do not solve.
The insidious part is that this leaves no visible trace for you. In your logs the request looks served. The bot received a challenge page instead of your content, but you see normal traffic and move on. On our three sites, Googlebot passed without trouble while GPTBot and ClaudeBot got the challenge: Google gets special treatment as a verified essential bot; AI bots do not always get the same luck.
The diagnosis: a user-agent matrix (10 minutes)
You don’t need special tools. You need curl and to look at the content, not the status code. This is the point almost everyone skips: the challenge can return 403, or even a “verifying browser” page that looks like a valid response. The only reliable thing is to compare what content actually arrived.
Run this for each bot:
curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
https://yoursite.com/ | grep -o "<title>[^<]*</title>"- If the
<title>that comes back is your real page title → that bot passes. - If it comes back “Just a moment…”, “Verifying you are human”, an empty title or an error → that bot is blocked.
Minimum bots to test:
| Bot | What it represents |
|---|---|
| ChatGPT-User | Live queries from ChatGPT users |
| Claude-User | Live queries from Claude users |
| Perplexity-User | Live queries from Perplexity users |
| OAI-SearchBot | The ChatGPT search index |
| PerplexityBot | The Perplexity index |
| Claude-SearchBot | The Claude index |
| GPTBot / ClaudeBot | Model training |
| Googlebot | Control: should always pass |
And one decision worth making on purpose: not all AI bots are the same. Training bots read your site to train models; you can legitimately block them if that’s your call. Search bots feed the indexes AIs cite from. And live-query bots are a person asking at that exact moment. Blocking those is closing the door on someone who was already asking about you. The right answer is almost never “block everything” or “allow everything”: it’s choosing by level.
The fix: two settings
From the Cloudflare dashboard, for each zone:
- Security → Settings → Browser Integrity Check: turn off.
- Security → Bots: if you have Bot Fight Mode or Super Bot Fight Mode, turn off JS Detection (or configure it so it doesn’t challenge the AI bots you decided to allow).
Via API (the verifiable and repeatable way):
# Browser Integrity Check → off
curl -X PATCH "https://api.cloudflare.com/client/v4/zones/ZONE_ID/settings/browser_check" \
-H "Authorization: Bearer YOUR_TOKEN" -H "Content-Type: application/json" \
--data '{"value":"off"}'
# JS Detection → off
curl -X PUT "https://api.cloudflare.com/client/v4/zones/ZONE_ID/bot_management" \
-H "Authorization: Bearer YOUR_TOKEN" -H "Content-Type: application/json" \
--data '{"enable_js": false}'Do you lose protection? Marginally. BIC stops naive scrapers; scanning traffic will keep coming and will keep getting 403/404 on what doesn’t exist. On our sites that junk is measured and reported separately, not hidden.
Verify it worked
Run the user-agent matrix again: every bot you decided to allow should now receive your real content. Over the following weeks, check your logs: you should start seeing requests from OAI-SearchBot, PerplexityBot and friends.
What nobody tells you: this can break again
Today’s fix doesn’t guarantee tomorrow’s state. Cloudflare changes defaults, new bots appear, someone opens the dashboard and flips a toggle without knowing what it did. On our sites the block was active with nobody asking for it: it came from the factory.
That’s why verification is not an event, it’s a habit. At nimboweb this runs every month inside the report: which AIs can read your page, how many live queries you received, and whether something changed again. If you’d rather do it yourself, this guide is all you need. If you’d rather never think about it again: the monthly report costs $24.990 CLP.
Is your site ready for AI?
Paste your URL and get a report in 24h with 5 concrete findings: whether AI bots can read you, what you need to be cited, and how your site compares to one optimized with the nimboweb Protocol.