Skip to content

How ready is your website for AI and search?

What robots.txt tells each search and AI crawler, and what else your site offers AI agents.

For example example.com, without https:// or a path. The check reads its robots.txt, llms.txt, one sitemap and the home page.

AI readiness

Ready to check

Enter an address and press Check.

  • robots.txt
  • llms.txt
  • Sitemap
  • Structured data (JSON-LD)

What robots.txt tells each search and AI crawler, and what else your site offers AI agents.

How to use

  1. Type the site you want to check, such as example.com, without https:// or a path.
  2. Press Check. The tool reads the site's robots.txt, llms.txt, one sitemap and home page.
  3. Read the checks: each says in plain words what was found and what to add or fix. The list below shows what robots.txt tells each known crawler.

How it works

Search engines and AI systems read a few public files before they use a site. robots.txt says which crawlers may fetch which paths, and may carry a Content-Signal line saying whether the content may be used for search, as AI answer input or for AI training. llms.txt is a short Markdown guide to the site for language models. A sitemap lists the pages, and JSON-LD on the home page tells machines in schema.org terms what the site is. WebMCP lets a page offer tools to an AI agent in the browser. None of the last four is required; together they make a site easy to find and use.

For each known crawler the tool applies the rules of RFC 9309, the standard for robots.txt: a crawler follows the group naming it, otherwise the group for *, and the longest matching rule wins, with Allow winning a tie. It asks only about the home page, "/". Example: if robots.txt has "User-agent: GPTBot" with "Disallow: /" and a "User-agent: *" group that allows everything, GPTBot is blocked by its own rules and every other crawler is allowed. A missing robots.txt means no limits; one that cannot be read because of a server error can make crawlers hold back, so it is flagged. Sources: RFC 9309; the Content Signals policy; llmstxt.org; sitemaps.org; schema.org.

The check reads four public files once from our server and makes no judgement about whether blocking an AI crawler is right: that is your choice, so the AI crawler line is only a note. It does not test rules for other paths or headers like X-Robots-Tag. WebMCP tools that scripts register after the page loads cannot be seen without running the page, so a missing marker is not proof that a site has none.

Sources

Sources and methods

Frequently asked questions

Should I block AI crawlers?

That is a business choice. Blocking training crawlers keeps your content out of model training; blocking AI search crawlers keeps you out of AI answers and their links. This tool only shows what your robots.txt currently says.

Does robots.txt keep content private?

No. It is a request that well-behaved crawlers honour, not access control. Anything truly private needs a login.

What is a Content-Signal line?

An optional robots.txt line such as Content-Signal: search=yes, ai-input=yes, ai-train=no. It states how the content may be used. Some crawlers read it; it does not block anyone by itself.

Do I need llms.txt?

It is a proposal, not a requirement, and no search engine promises to use it. It is cheap to publish and some AI tools read it, so it is flagged only when it exists but is malformed.

Why is my sitemap marked as not named in robots.txt?

The tool found it at /sitemap.xml, but robots.txt has no Sitemap: line. Adding the line lets every crawler find it without guessing.

Why can't it see my WebMCP tools?

WebMCP tools are usually registered by script after the page loads. The tool looks only for markers in the HTML, such as a form marked as a tool.

Send feedback