How crawlers find and apply the file
The Robots Exclusion Protocol started as an informal convention in 1994 and became RFC 9309 in 2022. Before fetching pages, a crawler requests /robots.txt from the root of the host. The file applies only to that scheme, host and port, so https://example.com/robots.txt says nothing about blog.example.com, which needs its own file.
The file is a list of groups. A group opens with one or more User-agent lines naming a crawler's product token, followed by Allow and Disallow rules whose values are path prefixes. A crawler obeys the one group that best matches its token, falls back to the * group when none does, and ignores the rest.
What each control writes
The output always opens with User-agent: * and the general rules. Allow crawling writes one Disallow line per path in your list, or an empty Disallow: when the list is empty, which permits everything. Block everything swaps the list for Disallow: /. A new path gets a leading slash unless it starts with / or *.
Each of the twelve listed crawlers has a three-way selector. Block appends a group for that token containing Disallow: /, Allow all appends one containing Allow: /, and Default writes nothing. The list holds Googlebot, Bingbot, the AI-related tokens GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot, CCBot, Applebot-Extended and Bytespider, and two commercial SEO crawlers. Per-bot path rules have to be typed into the output.
A crawl delay is written under the * group only, so a crawler with its own group does not inherit it. Sitemap URLs go at the end, one Sitemap line each. Typing into the output box marks it (edited), but changing any setting in the Rules panel afterwards regenerates the file and discards those edits.
How the tester reaches Allowed or Blocked
The tester applies the precedence rules of RFC 9309. It chooses the group first: a group matches when its token appears anywhere in the user agent you typed, ignoring case, so a full User-Agent header pasted from a log works as well as the bare word Googlebot. The longest matching token wins and * is the fallback.
Inside that group, a rule applies when it matches the start of the test path. An asterisk stands for any run of characters and a trailing $ pins the rule to the end, so Disallow: /*.pdf$ blocks /files/report.pdf but not /files/report.pdf?page=2. The longest matching rule wins, and Allow beats Disallow at equal length. Paths are case-sensitive: a rule for /admin/ does not cover /Admin/.
The result names the deciding line, for example Matched line 2: Disallow: /admin/ (group: *), or says No rule matches this path when the request is allowed by default. Nothing is fetched. To check a live file, paste its contents into the output box.
Mistakes that surface after upload
The costliest error is a staging file containing Disallow: / that ships to production. Crawling stops and pages fade from search results over the following weeks. Test the home page and a few important URLs against the exact text you plan to deploy.
Blocking stylesheet, script or image directories comes second. Search engines render pages to judge layout and mobile usability, and a page whose CSS or JavaScript is disallowed looks broken to them, so let crawlers fetch whatever the page needs to display.
- Disallow: /blog also blocks /blog-archive and /blog.html. Write /blog/ to cover only the directory.
- A Sitemap line needs an absolute URL with scheme and host. /sitemap.xml alone is not valid.
- The file name is lowercase robots.txt, served from the root and never from a subfolder.
AI tokens, crawl delay and server errors
The AI tokens come in two kinds. GPTBot, ClaudeBot and CCBot are crawlers that make their own requests. Google-Extended and Applebot-Extended are control tokens: Googlebot and Applebot still crawl for search, and the token tells the company not to use what they fetch for its AI models. Blocking Google-Extended does not change how a site appears in Google Search.
Crawl-delay is not part of RFC 9309: Googlebot ignores it and Bing honours it. The response code of the file itself matters too. A 404 means no restrictions, while a 5xx error obliges a compliant crawler to treat the whole site as disallowed until the file is reachable again.