Beamfar
Sign inStart the 14-day trial

Cloaking for AI crawlers and why it hurts LLM visibility

By the Beamfar team. Published . Updated . 7 min read.

Cloaking for AI crawlers means serving the bots of AI products a page that differs from the one a person gets, usually after checking the user agent. It is search cloaking aimed at a new reader, and it damages visibility in two ways: Google treats cloaking as spam, which can lower a site or remove it from results, and an AI answer built from a page that visitors never see says things the site does not show. Test your own sites by requesting a page with each crawler’s user agent and comparing the text, and verify real crawlers by the IP ranges their operators publish.

What it is, and why it is easy

A web server decides what to send from the request. The user agent is one signal, and the address the request comes from is another. Cloaking uses them to tell a bot from a visitor and answer each differently. Google defines the practice as “presenting different content to users and search engines” and gives the example of “inserting text or keywords into a page only when the user agent that is requesting the page is a search engine”.

The same trick works on AI products, which fetch pages for training, for search indexes and for a user’s question. OpenAI, Anthropic and Perplexity each document separate crawlers for those jobs: OpenAI lists OAI-SearchBot for search, GPTBot for training and ChatGPT-User for requests a user triggers; Anthropic lists ClaudeBot, Claude-SearchBot and Claude-User; Perplexity lists PerplexityBot and Perplexity-User. Each has a name that a script can match.

Security researchers have shown how simple it is. SPLX published a test in October 2025 in which a site returned one page to ordinary browsers and another to AI crawlers, and reported that it served the altered page to the crawlers behind OpenAI’s Atlas and ChatGPT and behind Perplexity; as reported, a simple user-agent check was enough to tell them apart. Their warning was that systems which treat retrieved pages as ground truth can be pushed to repeat misinformation, favour one candidate or one product, or damage a reputation.

Who does it, and why

Two groups. Some site owners, or SEO vendors working for them, try to feed AI products favourable claims that the visible site does not make. And attackers who have compromised a site inject spam or links and show them to bots only, so that the owner, who looks at the site in a browser, sees nothing; see How SEO spam hacks hide on WordPress sites. The second case is the one an agency meets.

Why it hurts visibility

Google’s rules cover the search side. The spam policies say that “sites that violate our policies may rank lower in results or not appear in results at all”. Google’s policy names search engines, not other companies’ bots, so the rule does not by itself reach a page that differs only for GPTBot. The link is Google’s own AI features: its guide says the best practices for SEO remain relevant for AI Overviews and AI Mode and that there are no additional requirements to appear in them. We read that as: what damages a site in Search damages it there too.

AI answers inherit what the bot was shown. An answer that cites your page and states a claim your visible page does not make is a liability for the client: a visitor who follows the citation finds something else. That is a trust problem for the brand, whatever any operator’s policy says.

No operator publishes a rule we could cite. The crawler pages of OpenAI, Anthropic and Perplexity that we read explain robots.txt controls and, for OpenAI and Perplexity, IP ranges; none states a penalty for serving a bot different content. It would be wrong to claim one. The risk to plan for is the loss of the trust above, a cleanup after a hack and the effect on Google.

User agents are a weak signal in both directions. In August 2025 Cloudflare reported that Perplexity used crawlers that did not declare themselves, with a generic browser user agent and addresses outside its published ranges; Perplexity disputed the finding. Whatever the truth, the lesson for site owners holds: a user-agent check can be fooled by a crawler that looks like a browser, and a disguised crawler would be served the visitor’s version. Operators that publish IP ranges (OpenAI and Perplexity do) give you the stronger way to recognise their real bots.

Check a client site

  1. Fetch the page as different readers and compare the text.
uas=(
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
  "ClaudeBot"
  "PerplexityBot"
)
for i in "${!uas[@]}"; do
  curl -s -A "${uas[$i]}" https://www.example.com/ > "page-$i.html"
done

Strip scripts and styles and compare the visible text, because timestamps and tokens make the raw HTML differ even on an honest site. Adjust the user-agent strings to the current ones in each operator’s documentation.

  1. Do it for the paths that matter: the home page, the pricing or product page, and any page that makes a claim.
  2. Read the server and CDN rules. Look for any rule or plugin that branches on the user agent, and for a CDN setting that challenges or alters responses for bots.
  3. Check your logs against the published ranges. Requests that claim to be GPTBot but come from other addresses are not OpenAI’s; the operator’s JSON list of ranges is the reference.
  4. Ask Google. Use Search Console’s URL Inspection to see what Googlebot received, as in the SEO spam guide.

If you find a difference you did not intend, treat the site as compromised until proven otherwise.

What to do instead

Show every reader the same content. Control access with robots.txt, which the operators’ pages describe as the control for their automatic crawlers (read each page for the user-triggered fetchers, which can differ), and let the search and user crawlers in if you want to be cited: see robots.txt mistakes that take a client site out of search. If a technical need makes you serve bots a different format (for example a pre-rendered page), keep the content identical, and say so in your care notes.

What outside monitoring can and cannot see

Beamfar’s Security checks read every page as a browser. For a site whose owner proved control of the domain with a DNS record and switched the crawler views on, they also request the home page and the next two pages as Googlebot, as a visitor who arrives from a search result and as four AI crawlers (GPTBot, ChatGPT-User, PerplexityBot and ClaudeBot), each time with Beamfar’s own name added to the user agent. Without that proof no request pretends to be a crawler. They compare the answers and report a difference as a possible problem with the evidence: another title and text, other scripts or links, spam vocabulary only the crawler sees, or a redirect to another site. Two browser requests are compared first, so a page that changes by itself is less likely to be reported. A refusal is not reported as cloaking: a firewall that blocks crawler-like requests is a notice.

The test cannot see a cloaker that checks the crawler’s address, because Beamfar does not request from Google’s or an operator’s published ranges; for Google, URL Inspection stays the check that counts. A difference is a lead, not proof of a hack, and a quiet result only means that nothing was seen by the checks that ran. Beamfar reads a plan-limited number of pages, is not a firewall and does not block or clean anything. It has been tried on test pages only and has no customers yet.

Questions

Is it cloaking to show AI crawlers a different page?

By Google’s wording, cloaking is showing users and search engines different content. A page that differs only for an AI company’s bot is not named there, but it is still deceptive, it is how hacked sites hide spam, and it makes AI answers repeat things the visible page does not say.

Does cloaking get a site banned from ChatGPT or Claude?

We found no published rule that says so. Plan for the other consequences: Google’s spam policy, a loss of trust when a citation does not match the page, and the cleanup if the cause was a hack.

How do I know a request is really from an AI crawler?

Compare its address with the ranges the operator publishes. OpenAI and Perplexity publish JSON lists, and a user-agent string alone can be copied by anyone.

Should I block AI crawlers instead?

That is a separate decision, made in robots.txt and independent of cloaking. Blocking and cloaking are different: blocking says no, cloaking says yes and then shows something else.

Sources

Read on . Standards and vendor pages change; confirm the current text before you rely on it.

Terms used in this guide

  • CrawlingCrawling is how a search engine fetches pages by following links and sitemaps. What affects it, why robots.txt and server errors change crawl rate and what crawl budget means.
  • IndexingIndexing is how a search engine stores a page it has crawled so it can be returned for queries. What decides whether a page is indexed and why crawled does not mean indexed.
  • robots.txtrobots.txt is a text file at the top of a site that tells crawlers which URLs they may fetch. It manages crawling, is not security and cannot reliably hide pages.
  • X-Robots-TagX-Robots-Tag is an HTTP response header that carries robots directives such as noindex, for pages and for files like PDFs. Why it is easy to miss in the page source.

Know first. Know why. Tell your client before they call.

Paste your client list. Both regions start checking.

Start the 14-day trialWhat Beamfar watches

No card for the trial.