Free, no sign-up
AI crawler access checker
Answer engines can only cite pages their crawlers may read. We read your robots.txt the way each AI crawler does and tell you which ones you let in.
This tool checks once. The GEO Agent works on it every day.
Meet the GEO Agent in Xicmo
Questions people ask
Should I block AI crawlers?
Block training crawlers if you do not want your content in model training, but keep the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) open. Those are the ones that let an answer engine link to you.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects pages for training OpenAI's models. OAI-SearchBot fetches pages so ChatGPT search can show and link them. You can allow one and block the other.
Is robots.txt enough?
It is the rule the main AI crawlers follow. An llms.txt file helps them further by pointing at your most useful pages in plain text.
More free tools
All tools48-hour free trial
Sign up today and run your whole marketing team free for 48 hours
- 600 credits to spend, no card needed
- Every agent: SEO, GEO, content, Reddit, X, email and more
- Up to 3 websites, the same as a Lite plan
- The Video Agent needs a subscription
The 48 hours start the moment you sign up.