Beamfar
Sign inStart the 14-day trial

What is robots.txt?

Last updated .

robots.txt is a plain text file served at the top level of a host, for example https://example.com/robots.txt. It lists, per crawler, which URL paths may or may not be requested. The Robots Exclusion Protocol is standardised in RFC 9309.

Definition
robots.txt
A text file at the top level of a host that tells compliant crawlers which URLs they may request; it controls crawling, not indexing.

What it does and does not do

It manages crawling. Google says it is not a mechanism for keeping a page out of Google: a disallowed URL linked from elsewhere may still be indexed without its content. RFC 9309 states that the rules are not a form of access authorization. To keep a page out of search use noindex on a crawlable page; to keep it private use authentication.

How crawlers read errors

A 4xx response means no file, so no restrictions. A 5xx response or a network failure means the crawler must assume everything is disallowed; Google stops crawling for the first 12 hours, then uses its cached copy for up to 30 days. Google caches the file for up to about 24 hours and ignores content beyond 500 KiB.

An example

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

This allows everything except paths that begin with /admin/ and declares the sitemap. The dangerous version is Disallow: /, which blocks the whole site and is often left over from staging. Rules match by path prefix, * and $ are supported inside a path, and when several groups name a crawler it obeys only the most specific one, so a group for one crawler must repeat any rule from the general group that should still apply to it.

What to check

That production does not serve Disallow: /, that the file returns 200, that the Sitemap: line points to the right host and that no rule blocks the CSS and JavaScript pages need.

Sources

Read on . Standards and vendor pages change; confirm the current text before you rely on it.

Guides that use this term

  • NoindexNoindex is a robots directive, set in a meta tag or header, that tells search engines not to keep a page in their index. How Google reads it and its common failure.
  • CrawlingCrawling is how a search engine fetches pages by following links and sitemaps. What affects it, why robots.txt and server errors change crawl rate and what crawl budget means.
  • SitemapAn XML sitemap lists the URLs a site wants search engines to know about. Google’s size limits, how lastmod is used and what belongs in the file.

Know first. Know why. Tell your client before they call.

Paste your client list. Both regions start checking.

Start the 14-day trialWhat Beamfar watches

No card for the trial.