Beamfar
Sign inStart the 14-day trial

What is the X-Robots-Tag header?

Last updated .

X-Robots-Tag is an HTTP response header that carries the same directives as the robots meta tag, for example X-Robots-Tag: noindex. Google’s documentation notes that a response header can be used for non-HTML resources such as PDFs, video files and images, and it works for pages as well.

Definition
X-Robots-Tag
An HTTP response header that carries robots directives such as noindex, usable for any file type and invisible in the page source.

Why it hides

The header is set by the web server, a CDN or an application, not by the page’s HTML. Viewing the source shows nothing. A server rule that adds X-Robots-Tag: noindex to every response for a staging host can stay in place after the site is moved to production, and the pages look perfect while search engines drop them.

How to see it

Request the headers: curl -sI https://example.com/page, or read the response headers in the browser’s network panel, or inspect the URL in Search Console. Check pages and file types separately.

An example

Google’s documentation shows how to keep PDF files out of search with an Apache rule: <Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </Files>. To address one crawler, put its name in front of the directive: X-Robots-Tag: googlebot: noindex. Because the header lives in server or CDN configuration, search for it in both places when a page is unexpectedly missing from search, and test it with curl -sI.

The header can also carry other rules, such as nosnippet or unavailable_after. For page-level control in HTML the meta tag is usually simpler; the header is the right tool when the resource is not HTML or when one server rule must cover many files.

What to check

That no response from a production host carries noindex, apart from pages that are meant to carry it, and that a CDN or server rule has not added it everywhere.

Sources

Read on . Standards and vendor pages change; confirm the current text before you rely on it.

Guides that use this term

  • NoindexNoindex is a robots directive, set in a meta tag or header, that tells search engines not to keep a page in their index. How Google reads it and its common failure.
  • robots.txtrobots.txt is a text file at the top of a site that tells crawlers which URLs they may fetch. It manages crawling, is not security and cannot reliably hide pages.
  • IndexingIndexing is how a search engine stores a page it has crawled so it can be returned for queries. What decides whether a page is indexed and why crawled does not mean indexed.

Know first. Know why. Tell your client before they call.

Paste your client list. Both regions start checking.

Start the 14-day trialWhat Beamfar watches

No card for the trial.