Sign up today and explore Xicmo free for 48 hours600 credits, no card
Xicmo
Get started

Free, no sign-up

Robots.txt generator

Pick what search engines and AI crawlers may read, list the paths that should stay private, and copy a robots.txt that does exactly that.

Keep crawlers out of
Block these AI crawlers

Unticked crawlers stay allowed. Keep the search ones open if you want AI answers to cite you.

robots.txt
# robots.txt generated with Xicmo's free tool
User-agent: *
Disallow: /admin/
Disallow: /api/

Generate robots.txt, sitemap.xml and llms.txt from your real site

  • robots.txt, sitemap.xml and llms.txt generated from your real site
  • Fixes for crawl errors Google reports, as pull requests
  • Alerts when a deploy blocks pages you want indexed
See Xicmo plansTry it free for 48 hours

This tool checks once. The SEO Agent works on it every day.

Meet the SEO Agent in Xicmo

See what it does

Questions people ask

Can robots.txt remove a page from Google?

No. It stops crawling, not indexing: a blocked page can still show up from links to it. To keep a page out of results, let it be crawled and add a noindex tag.

Should I block /login and /cart?

Pages with nothing to rank are better left crawlable with a noindex tag, so Google can read that tag. Block paths that waste crawl budget, like internal search results.

More free tools

All tools

48-hour free trial

Sign up today and run your whole marketing team free for 48 hours

  • 600 credits to spend, no card needed
  • Every agent: SEO, GEO, content, Reddit, X, email and more
  • Up to 3 websites, the same as a Lite plan
  • The Video Agent needs a subscription

The 48 hours start the moment you sign up.

Start your free trial See plans