What robots.txt is for
robots.txt tells crawlers which URLs they may request. It sits at the top level of a host, for example https://example.com/robots.txt, and it applies only to that host, protocol and port: a subdomain needs its own file. The Robots Exclusion Protocol was standardised as RFC 9309 in 2022.
Two points of principle prevent half the mistakes:
- It manages crawling, not indexing. Google’s documentation says it is not a mechanism for keeping a page out of Google. A disallowed URL that is linked from other sites can still be indexed, shown with the URL and no content. To keep a page out of search, use
noindexand let the page be crawled. - It is not security. RFC 9309 says the rules are not a form of access authorization, and a file that lists private paths advertises them. Use authentication for anything private.
The mistakes
1. The staging file reaches production
User-agent: *
Disallow: /This blocks every compliant crawler from the whole site. It is correct on staging and a disaster on production. It arrives with a file copy, a deployment that includes the whole web root or a CMS setting.
2. Blocking the files the page needs
Disallowing /wp-content/, /assets/ or /static/ stops Google fetching the CSS and JavaScript it needs to render the page. Pages may be judged on a broken rendering. Block crawl traps, not the resources the page needs.
3. Relying on Disallow to hide a page
A page disallowed in robots.txt cannot be crawled, so Google never sees a noindex on it. People add both and wonder why the page stays in results. If the aim is removal from search, remove the disallow and keep the noindex until the page drops out.
4. Wrong wildcards and anchors
Google supports * (any sequence of characters) and $ (end of the URL). Disallow: /*.pdf$ blocks PDFs. Disallow: /blog blocks /blog, /blog/, and also /blog-archive and /blogger, because the match is a prefix. Test the rule with real URLs before it ships.
5. Trusting rule order
Rules do not apply in order of appearance. RFC 9309 says the most specific match, meaning the one with the longest path, wins. A broad Disallow: /shop/ and a narrow Allow: /shop/sale/ give a different result from what you expect if you read top to bottom. When two rules conflict in specificity, Google uses the least restrictive one.
6. The file returns an error
What the crawler does depends on the response, and the result is surprising:
| Response for /robots.txt | What a compliant crawler does |
|---|---|
| 200 with a valid file | Follows the rules |
| 4xx (except 429 for Google) | Treats it as no file: everything may be crawled |
| 5xx or network failure | RFC 9309: assume everything is disallowed. Google stops crawling the site for the first 12 hours, then uses its last cached copy for up to 30 days |
So a deploy that leaves /robots.txt returning a server error can halt crawling, and a redirect or rewrite rule that turns it into a 404 page removes every restriction, including ones you wanted.
7. A huge or oversized file
RFC 9309 requires crawlers to parse at least 500 kibibytes. Google ignores content beyond 500 KiB. Keep the file small.
8. Caching hides a fix
Google caches the file for up to about 24 hours. A fixed file may not take effect for a day. In Search Console the robots.txt report shows the version Google last fetched.
9. Different files on different hosts
www, the bare domain, a shop subdomain and the CDN host may each serve a different file, or none. Check each host that serves pages. If http and https serve different content, the rules apply to each separately.
10. A missing sitemap line
The Sitemap: line is not required, but it helps crawlers find the sitemap, and a stale line pointing to a staging host sends them to the wrong place.
A safe way to edit
- Keep the file in version control, with a note of who owns it.
- Make the content depend on the environment automatically, not by copying files: staging serves the blocking file, production serves the real one.
- Test changed rules against a list of important URLs (home, a product, a blog post, a PDF, a CSS file). Search Console’s robots.txt report and any parser built on RFC 9309 will do.
- After release, fetch the live file from outside and read it.
- Keep a golden copy and compare the live file to it on a schedule. A one-line change in a file is easy to miss and costly.
A minimal production file
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /search
Sitemap: https://example.com/sitemap.xmlBlock pages that waste crawling (search results, carts) and nothing else. If you are not sure a rule is needed, leave it out.
How Beamfar helps
Beamfar reads /robots.txt for each monitored page’s host, applies the RFC 9309 rules for Googlebot and for all crawlers to the page’s path, and raises an SEO-risk alert when robots.txt starts blocking a page it did not block before. The alert is sent after a second sighting, so one bad response does not wake anyone. It checks the pages you monitor, not every URL on the site.
Questions
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing. A disallowed URL that is linked from elsewhere can still appear in results without its content. Use noindex on a crawlable page to keep it out of search.
What happens if robots.txt returns a 500 error?
Under RFC 9309 a crawler must assume that everything is disallowed. Google stops crawling the site for the first 12 hours of such errors and then relies on its cached copy of the file for up to 30 days.
What happens if robots.txt returns a 404?
A compliant crawler treats a 4xx response (other than 429 for Google) as no robots.txt, which means no restrictions on crawling.
How long does a robots.txt change take to apply?
Google caches the file for up to about 24 hours, so allow a day. The robots.txt report in Search Console shows which version Google last fetched.
Sources
Read on . Standards and vendor pages change; confirm the current text before you rely on it.
- RFC 9309, Robots Exclusion Protocol (www.rfc-editor.org)
- Google Search Central, How Google interprets the robots.txt specification (developers.google.com)
- Google Search Central, Introduction to robots.txt (developers.google.com)