# AI readiness checker > How ready is your website for AI and search? What robots.txt tells each search and AI crawler, and what else your site offers AI agents. Check a website for AI and search readiness: robots.txt rules for GPTBot, ClaudeBot and Googlebot, llms.txt, sitemap, JSON-LD and WebMCP markers. - Page: https://how.box/ai-readiness/ - Uses our server; server limits: 10 requests/minute per visitor, 150 requests/day per visitor, 40 page/minute per visitor, 10 requests/minute per target, 10 requests/second per global. - License: HowToolbox's own text and data (titles, explanations, FAQs, tool descriptions and schemas, this file) are CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/); AI training and answers are welcome. Attribute "HowToolbox (how.box)" with a link to the page used. Third-party data keeps its own terms; the name and logo are not licensed. Publisher: HowToolbox, contact@how.box. Terms: https://how.box/about/terms/ ## Inputs Answer link: https://how.box/ai-readiness/?h={h} - `h` (text): Public site (host name) to check; the check runs when the page opens. Example: `example.com`. Example answer: https://how.box/ai-readiness/?h=example.com The page computes the answer in the browser: read results from the agent tools below, not from the page HTML. ## Agent tools - `ai_readiness_check_site`: Checks how ready a public website is for search engines, AI crawlers and agents, from four public files read once by our server: robots.txt, llms.txt, one sitemap and the home page. - Page only: runs on this page (WebMCP), not over MCP. ## How it works Search engines and AI systems read a few public files before they use a site. robots.txt says which crawlers may fetch which paths, and may carry a Content-Signal line saying whether the content may be used for search, as AI answer input or for AI training. llms.txt is a short Markdown guide to the site for language models. A sitemap lists the pages, and JSON-LD on the home page tells machines in schema.org terms what the site is. WebMCP lets a page offer tools to an AI agent in the browser. None of the last four is required; together they make a site easy to find and use. For each known crawler the tool applies the rules of RFC 9309, the standard for robots.txt: a crawler follows the group naming it, otherwise the group for *, and the longest matching rule wins, with Allow winning a tie. It asks only about the home page, "/". Example: if robots.txt has "User-agent: GPTBot" with "Disallow: /" and a "User-agent: *" group that allows everything, GPTBot is blocked by its own rules and every other crawler is allowed. A missing robots.txt means no limits; one that cannot be read because of a server error can make crawlers hold back, so it is flagged. Sources: RFC 9309; the Content Signals policy; llmstxt.org; sitemaps.org; schema.org. The check reads four public files once from our server and makes no judgement about whether blocking an AI crawler is right: that is your choice, so the AI crawler line is only a note. It does not test rules for other paths or headers like X-Robots-Tag. WebMCP tools that scripts register after the page loads cannot be seen without running the page, so a missing marker is not proof that a site has none. ## FAQ **Should I block AI crawlers?** That is a business choice. Blocking training crawlers keeps your content out of model training; blocking AI search crawlers keeps you out of AI answers and their links. This tool only shows what your robots.txt currently says. **Does robots.txt keep content private?** No. It is a request that well-behaved crawlers honour, not access control. Anything truly private needs a login. **What is a Content-Signal line?** An optional robots.txt line such as Content-Signal: search=yes, ai-input=yes, ai-train=no. It states how the content may be used. Some crawlers read it; it does not block anyone by itself. **Do I need llms.txt?** It is a proposal, not a requirement, and no search engine promises to use it. It is cheap to publish and some AI tools read it, so it is flagged only when it exists but is malformed. **Why is my sitemap marked as not named in robots.txt?** The tool found it at /sitemap.xml, but robots.txt has no Sitemap: line. Adding the line lets every crawler find it without guessing. **Why can't it see my WebMCP tools?** WebMCP tools are usually registered by script after the page loads. The tool looks only for markers in the HTML, such as a form marked as a tool. More: [HowToolbox index](https://how.box/llms.txt) · [Agent Skill](https://how.box/skill.md) · [agent terms](https://how.box/about/agents/)