AI search

AI crawlers

Retrievability comes before citation. This is what each answer engine's crawler can reach, how often it comes back, and how much of what it crawls ends up quoted.

AI crawler requests (30d)
49,780
Bots allowed
5 of 8
Avg retrieval yield
19.6%
llms.txt
Missingpublish recommended

AI crawl volume

Daily requests from answer-engine crawlers across the site.

Retrieval yield by bot

Pages cited as a share of pages crawled.

  1. GPTBot23.7%
  2. OAI-SearchBot22.8%
  3. ClaudeBot15.8%
  4. PerplexityBot29.7%
  5. Applebot-Extended5.8%

Crawl posture

What robots.txt, llms.txt and the sitemap currently tell answer engines.

  • robots.txt

    User-agent: GPTBot / Allow: /

    Full site exposed to OpenAI training and search crawlers.

  • robots.txt

    User-agent: Google-Extended / Disallow: /

    Blocks Gemini grounding — we are cited 12 times despite the block, via third parties.

  • robots.txt

    User-agent: CCBot / Disallow: /

    Common Crawl feeds several smaller engines; blocking reduces long-tail reach.

  • llms.txt

    Missing

    No llms.txt published — engines have no curated map of canonical pages.

  • sitemap.xml

    Present, 412 URLs

    Last updated 2 days ago; all key pages included.

Blocked but valuable

High-traffic pages engines cannot currently retrieve.

  • /seo-singapore18,420 clicks
  • /services/seo9,840 clicks
  • /singapore/seo-agency4,210 clicks
  • /resources/seo-pricing-singapore7,620 clicks

Crawled but never cited

Retrieved repeatedly, quoted never — a passage-quality problem.

  • /services/local-seo8,989 clicks
  • /services/link-building6,709 clicks
  • /services/content-marketing437 clicks
  • /services/ai-search-optimisation8,644 clicks
  • /singapore/digital-marketing-agency5,710 clicks
  • /malaysia/seo-agency7,677 clicks

8 rows

Trend

GPTBot

OpenAI

Allowed18,4203127423.7%1d ago

PerplexityBot

Perplexity

Allowed11,2902968829.7%2d ago

OAI-SearchBot

OpenAI

Allowed9,3102686122.8%2d ago

ClaudeBot

Anthropic

Allowed7,6402413815.8%1d ago

Applebot-Extended

Apple

Allowed3,12015495.8%2d ago

Google-Extended

Google

Blocked0012—121d ago

CCBot

Common Crawl

Blocked004—121d ago

Bytespider

ByteDance

Blocked000—121d ago