Your CDN may be hiding your client from every AI engine

The single most common finding in our readiness diagnostics is not a content problem. It is that the client's edge is refusing the crawlers that feed AI answers, and nobody chose that.

How it happens

Bot management has become a default rather than a decision. A managed bot-fighting mode, a firewall rule written years ago against scrapers, an aggressive rate limit — any of these can return a challenge or a 403 to GPTBot or ClaudeBot while serving Googlebot perfectly. The site ranks fine. It is simply invisible to the systems increasingly answering the question first.

The crawlers to check, individually

They are separate agents with separate purposes, and a blanket allow or block is almost never what the client actually wants. Blocking the training crawlers while allowing the search crawlers is a coherent position; blocking everything by accident is not.

The second silent failure

Even when the crawler gets through, it does not run JavaScript. No major AI crawler does. A client on a client-rendered framework whose product copy arrives after hydration is serving those crawlers an empty shell — and their analytics will show the crawler visiting successfully.

A ten-minute audit

1. Fetch a key page with each bot's user-agent from outside your own network. Record the status code, not just whether something came back.
2. Fetch it again with JavaScript disabled and read what is actually in the HTML.
3. Check the CDN's bot rules for a managed category that includes AI crawlers.
4. Check robots.txt for a blanket rule somebody added in 2023.