Robots.txt tester

What robots.txt actually controls

A robots.txt file sits at the root of a site and tells crawlers which addresses they may fetch. That is all it does. It is a request about crawling, not an instruction about indexing, and the gap between those two words is the most expensive misunderstanding in technical SEO.

Google's own documentation is blunt about it: a page blocked in robots.txt can still appear in search results if other pages link to it, because Google can learn the address exists without ever fetching it. It simply has nothing to show underneath. If you need a page kept out of search, let it be crawled and use a noindex tag — which Google can only see if it is allowed to fetch the page in the first place.

What this tester does

It fetches the live robots.txt, reads it with the same parser our crawler uses, and tells you whether Googlebot is allowed to fetch the address you typed. The parsing follows what Google honours rather than the original 1994 standard: user-agent grouping, Allow beating Disallow when it is more specific, * wildcards and $ end-of-match anchors.

The faults worth knowing about

• A 5xx on robots.txt stops the crawl. Google documents that when it finds a robots.txt and cannot fetch it, it stops crawling the site for the first 12 hours. A broken file is worse than no file.
• Disallow: / left over from staging. It blocks the whole site, and it is the single most common way a new site stays invisible for weeks.
• Blocking CSS or JavaScript. Google renders pages. A stylesheet it cannot fetch is a page it judges half-built.
• Crawl-delay. Google does not support the field and ignores it. Our crawler obeys it.
• Over 500 KiB. Google reads the first 500 KiB and ignores the rest.

Next: how to run a full SEO audit, or check your sitemap.