Beamfar
Sign inStart the 14-day trial

Noindex after a deploy: how a client site disappears from Google

By the Beamfar team. Published . Updated . 5 min read.

The most expensive one-line mistake in agency work is a noindex that was right on staging and goes live with the site. Nothing looks wrong: the pages load, the design is perfect and the client is delighted. A few weeks later their organic traffic is gone, and nobody can say when it started.

What noindex does

A noindex rule tells search engines that support it not to keep a page in their index. Google documents two ways to send it:

  • A meta tag in the page’s head: <meta name="robots" content="noindex">. Use name="googlebot" to target only Google.
  • An HTTP response header: X-Robots-Tag: noindex. This is the way to cover files that are not HTML, such as PDFs and images, and it also works for pages.

Google’s documentation adds two points that matter for diagnosing a problem. First, the page must not be blocked by robots.txt, because Googlebot has to fetch the page to see the rule. Second, Google acts on the rule only after it recrawls the page, and the documentation says that can take months for a page that is rarely visited. It is the reverse for the problem you are about to have: pages will fall out of the index at the pace Google recrawls them, and they will not come back until they are recrawled again after you remove the tag.

How it reaches production

The staging site was told to stay out of Google, correctly. The tag survives into production because of one of these:

  • A site-wide setting. WordPress has a “Discourage search engines from indexing this site” option. Copying a database or an export from staging to production copies the setting too.
  • A template or layout file that adds noindex for all environments, with a condition that is wrong on the new host.
  • An environment variable such as APP_ENV=staging carried over into production.
  • A server rule or CDN setting that adds X-Robots-Tag to every response. This is invisible in the page source, which is why people miss it.
  • A platform default. Some site builders and hosting tools add noindex on temporary preview URLs; moving the domain does not always remove it.
  • A password-protected launch that was lifted, but with the tag left in.

Why it is hard to notice

  • The site works. No monitor that checks for “page loads and shows the expected text” will complain.
  • Rankings fall slowly, with the speed of recrawl, so the loss does not coincide with the deploy.
  • The tag may be on one template only. The home page is fine and the blog is not.
  • The header version cannot be seen by viewing the source.

Check for it in under two minutes

For a page, check both places:

  1. View the source and search for robots. Note every <meta name="robots"> and <meta name="googlebot">. There should be none, or one that says index.
  2. Read the response headers: curl -sI https://example.com/ | grep -i x-robots-tag. There should be nothing.
  3. In Google Search Console, run the URL Inspection tool on the page and read “Indexing allowed?” and “Page indexing”.
  4. Check the Pages report for a rising number of “Excluded by ‘noindex’ tag” URLs.

Check at least one URL for every template: home, a service page, a blog post, a product, a category, a PDF.

A pre-deploy and post-deploy routine

Before the deploy

  • Name the environments in the brief: which hosts must be noindex (staging, preview) and which must not (production).
  • Make the noindex rule depend on the host name that serves the page, not on a copied setting.
  • Put “search engine visibility” in the launch checklist as an item with a person’s name.

Immediately after

During the following weeks

  • Repeat the check after each release and after each plugin or theme update.
  • Watch the indexed page count in Search Console. A fall of more than a few percent in a week deserves a look.

If it has already happened

  1. Remove the tag or header at its source, not only on one page. Search the code, the settings, the server and the CDN.
  2. Check the live pages again from outside.
  3. In Search Console, inspect the key URLs and use “Request indexing” for the most important ones. Resubmit the sitemap.
  4. Do not block the pages in robots.txt “to be safe”: that stops Google from seeing that the tag is gone.
  5. Tell the client what happened, how long the pages were affected, and what now checks for it. Expect recovery to follow the recrawl, not your fix.

What not to rely on

  • Robots.txt is not a way to keep a page out of search. Google says it is not a mechanism for keeping a page out of Google, and a disallowed URL linked from elsewhere can still be indexed without its content. Use noindex for that, and let Google crawl the page.
  • Noindex in robots.txt. Google does not support it.
  • A visual check. The page looks fine in every case.

How Beamfar helps

Beamfar looks for this risk on every monitored site. It raises an alert when a page becomes noindex, either as a meta tag or a header, when robots.txt starts blocking it, or when its canonical moves to another host. To avoid false alarms it alerts after a second sighting, not on a single reading. It checks the pages you add as monitored URLs, so add one per template; it does not replace Search Console’s view of what Google has indexed.

Questions

Does noindex remove my page from Google immediately?

No. Google applies it when it next crawls the page; for a rarely visited page its documentation says that can take months. Recovery after you remove the tag also depends on the next crawl.

Can robots.txt and noindex be used together?

Not usefully. If robots.txt blocks a page, Google cannot fetch it and never sees the noindex rule, so the URL may stay indexed without content if it is linked elsewhere.

Where can I see an X-Robots-Tag header?

In the response headers, not in the page source. Use curl -I, the browser’s network panel or Search Console’s URL Inspection tool.

How often should I check indexability on client sites?

After every release, plugin or theme update, host move and CMS settings change, and at least monthly for a care plan. The failure comes from changes, so check after changes.

Sources

Read on . Standards and vendor pages change; confirm the current text before you rely on it.

Terms used in this guide

  • NoindexNoindex is a robots directive, set in a meta tag or header, that tells search engines not to keep a page in their index. How Google reads it and its common failure.
  • X-Robots-TagX-Robots-Tag is an HTTP response header that carries robots directives such as noindex, for pages and for files like PDFs. Why it is easy to miss in the page source.
  • robots.txtrobots.txt is a text file at the top of a site that tells crawlers which URLs they may fetch. It manages crawling, is not security and cannot reliably hide pages.
  • IndexingIndexing is how a search engine stores a page it has crawled so it can be returned for queries. What decides whether a page is indexed and why crawled does not mean indexed.

Know first. Know why. Tell your client before they call.

Paste your client list. Both regions start checking.

Start the 14-day trialWhat Beamfar watches

No card for the trial.