Beamfar
Sign inStart the 14-day trial

What is noindex?

Last updated .

Noindex is a rule that tells search engines not to include a page in their index. It is sent in a meta tag, <meta name="robots" content="noindex">, or in an HTTP response header, X-Robots-Tag: noindex. Google documents both.

Definition
Noindex
A robots directive, sent as a meta tag or an X-Robots-Tag header, that tells search engines not to keep a page in their index.

How Google applies it

Google has to crawl the page to see the rule, so the page must not be blocked by robots.txt. Google acts after it next crawls the page, which its documentation says may take months for pages that are rarely visited. Google does not support noindex written in robots.txt.

The usual failure

A staging site is correctly set to noindex, and the setting travels to production with a database copy, a template or a server rule. The site works for visitors and slowly disappears from search.

A quick audit

Run curl -s https://example.com/ | grep -i 'name="robots"' to see the meta tag and curl -sI https://example.com/ | grep -i x-robots-tag to see the header. Run both for production and for staging: staging should show noindex, production neither. Put the two commands in the release checklist, because the setting that protects staging is the one most often copied to production by mistake.

Noindex removes a page from search results but does not hide it: anyone with the address can still open it, and a page that must stay private needs authentication, not a meta tag. When the rule is removed, the page returns to the index after its next crawl, not instantly.

What to check

The meta tags in the source, the response headers (which the source does not show), and Search Console’s URL Inspection and Pages report, for at least one URL per template, after every release.

Sources

Read on . Standards and vendor pages change; confirm the current text before you rely on it.

Guides that use this term

  • X-Robots-TagX-Robots-Tag is an HTTP response header that carries robots directives such as noindex, for pages and for files like PDFs. Why it is easy to miss in the page source.
  • robots.txtrobots.txt is a text file at the top of a site that tells crawlers which URLs they may fetch. It manages crawling, is not security and cannot reliably hide pages.
  • IndexingIndexing is how a search engine stores a page it has crawled so it can be returned for queries. What decides whether a page is indexed and why crawled does not mean indexed.

Know first. Know why. Tell your client before they call.

Paste your client list. Both regions start checking.

Start the 14-day trialWhat Beamfar watches

No card for the trial.