Problem · AI search
AI crawlers can’t reach our site. Decide on training and search separately.
Some sites block AI crawlers without knowing, through robots.txt, a CDN or a security tool. Blocking training crawlers is a business choice; blocking search crawlers removes you from answers.
Is this you?Signs we usually see
- AI assistants say they can’t access your site
- Your pages are never cited, even for your own brand
- robots.txt blocks bots you don’t recognise
- A CDN or security tool challenges bots
01 / Quick checks
Three checks you can run today
Each takes minutes. Together they tell you whether this page is about your problem.
- Read robots.txtLook for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended rules.
- Check your CDN settingsSome CDNs and firewalls now block AI crawlers by default.
- View the raw HTMLMany AI crawlers don’t run JavaScript. Is your text in the page source?
Crawler-purpose access register
Separate search retrieval, training and user-triggered fetch controls before changing robots or edge rules.
| Situation to check | Proposed scope | Boundary |
|---|---|---|
| Read robots.txt | Look for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended rules. A blocked legitimate request at the CDN requires a different fix from a deliberate robots preference. | Allowing a crawler does not guarantee inclusion; opening every bot can conflict with the site's own access policy. |
| Check your CDN settings | Some CDNs and firewalls now block AI crawlers by default. | Keep an unchanged comparison sample so this check can distinguish the proposed cause from unrelated variation. |
| Blanket bot blocking | Rules that block all AI user agents, search ones included. | Treat this as a hypothesis until the affected URL or account evidence supports it. |
| Ready to commission a fix | Validate the identified user agent against official documentation and retain the request response and chosen policy. | [CONFIRM: access, scope, owner, fee and review date before work begins.] |
Crawler-purpose access register
Illustrative blank specification. This is a diagnostic structure, not a client result.
- Affected population
- Separate search retrieval, training and user-triggered fetch controls before changing robots or edge rules.
- Decisive observation
- A blocked legitimate request at the CDN requires a different fix from a deliberate robots preference.
- Alternative explanation
- Allowing a crawler does not guarantee inclusion; opening every bot can conflict with the site's own access policy.
- Acceptance evidence
- Validate the identified user agent against official documentation and retain the request response and chosen policy.
- Supporting check
- Many AI crawlers don’t run JavaScript. Is your text in the page source?
- Approval
- [CONFIRM: accountable owner and authorised next step.]
What the first handover should make clear
- Separate search retrieval, training and user-triggered fetch controls before changing robots or edge rules.
- A blocked legitimate request at the CDN requires a different fix from a deliberate robots preference.
- Validate the identified user agent against official documentation and retain the request response and chosen policy.
- The original observations, unresolved questions and any client access needed for the next scope.
02 / Likely causes
What usually causes it
In the order we usually check them. Most cases have more than one.
- Blanket bot blockingRules that block all AI user agents, search ones included.
- CDN or firewall defaultsBot protection that blocks or challenges crawlers.
- JavaScript-only contentNothing to read without rendering.
- Login or cookie wallsContent behind consent or sign-in.
- Server errors for botsRate limits returning errors to crawlers.
03 / What we do
How we fix it
Four steps, in order. You see the evidence and the plan before anything changes on your site.
- Step 01Decide the policyTraining access vs search access, bot by bot, documented.
- Step 02Fix accessrobots.txt, CDN and firewall rules aligned.
- Step 03Make pages readableKey content in the HTML.
- Step 04VerifyServer logs and test fetches by each bot.
04 / Services
Services that fix it
Starting prices in USD. TopClick is not VAT-registered. Scope and price in writing before work starts.
TopClick is not VAT-registered
AI visibility audit ↗A one-off baseline of where you stand in AI answers, with a prioritised plan.From $[X] /projectRequest
Technical SEO ↗The causes behind crawl, rendering and indexation problems, found and fixed.From $[X] /projectRequest
LLM optimization (LLMO) ↗Get described correctly by language models, with and without live search.From $[X] /monthRequest
AI Search & GEO ↗Become a source ChatGPT, Perplexity and Google AI Overviews cite.From $[X] /monthRequest
05 / Questions
Questions about this problem
01Does blocking GPTBot remove us from ChatGPT?
GPTBot is used for training. ChatGPT search uses OAI-SearchBot. They are separate decisions.
02Does Google-Extended affect Google Search?
No. Google-Extended controls use for training Gemini models; it doesn’t affect how Google Search crawls or ranks you.
03Where do we start?
With the free snapshot: we check your site for this problem, confirm the likely cause and send a written first scope with a price.
Start here
Check your site for this
- 1Send your site — we confirm the cause and where we would start
- 2We check search and AI visibility
- 3PDF by email: five website problems and AI visibility, within two business days of receiving your website URL
Prefer to talk?Book a 20-min callhello@topclick.expert
Request receivedWe'll reply within one business day. Want to talk sooner? Book a call in the next tab.See what happens next →
20-min intro callVIDEO CALL · FREEA strategist looks at your site with you and suggests where to start. No pitch.
Times in your time zone · your local time
Enter a valid email address
We couldn’t send this. Please try again or email hello@topclick.expert.
RequestedWe’ll confirm the time by email within [X] business hours.How to prepare for the call →