Fixing Preview Deploys Leaking Into Search Results

A search for a product's own documentation returns, in third place, docs-git-feature-pricing-acme.vercel.app/pricing/ — a preview deploy from a pull request merged four months ago, with outdated prices. Preview deploys are one of the best features of modern static hosting, and one of the easiest ways to put duplicate, stale copies of a site in front of search engines. Every preview is a complete copy of the site on a public URL, and it takes one link in a public place for a crawler to find it.

This guide removes indexed preview and staging URLs for a documentation site that used preview deploys on Vercel, Netlify and Cloudflare Pages over its history, and prevents new ones being indexed on each host. It belongs to Preview Environments for Pull Requests; the general robots and noindex setup is covered in robots.txt and noindex for Preview and Staging.

Step 1: Find What Is Indexed

Search with site: for each preview domain pattern, and check Search Console for the production property's "Duplicate, Google chose different canonical than user" entries, which list preview URLs chosen over production pages:

The 47 were the real damage: for those pages, search showed the preview copy instead of the production page.

Step 2: Find How They Were Discovered

The preview URLs had been linked from three public places: the repository was public, and the deploy bot's comments on pull requests linked every preview; the docs' public issue tracker contained preview links in discussions; and the staging branch's URL was in the project README. Search engines crawl GitHub, so every pull request comment was a discovery path.

Making previews unlinkable was not realistic — reviewers need the links. The fix had to be on the previews themselves.

Step 3: Send noindex From Every Non-Production Host

The reliable mechanism is an X-Robots-Tag: noindex header on every response from a preview host, set by host-based rules so production is never affected.

Vercel (vercel.json), using has on the host:

{
  "headers": [
    { "source": "/(.*)", "has": [{ "type": "host", "value": "(?<host>.*\\.vercel\\.app)" }],
      "headers": [{ "key": "X-Robots-Tag", "value": "noindex, nofollow" }] }
  ]
}

Vercel also adds X-Robots-Tag: noindex to preview deployments by default on its own domains for many projects; the explicit rule makes it independent of that default and covers branch aliases.

Netlify (netlify.toml), using deploy contexts:

[context.deploy-preview]
  [[context.deploy-preview.headers]]
    for = "/*"
    [context.deploy-preview.headers.values]
      X-Robots-Tag = "noindex, nofollow"

Cloudflare Pages has no context-specific _headers, so the build writes a different _headers file depending on the branch, using CF_PAGES_BRANCH:

if [ "$CF_PAGES_BRANCH" != "main" ]; then printf '/*\n  X-Robots-Tag: noindex, nofollow\n' >> dist/_headers; fi

Cloudflare Pages also sends X-Robots-Tag: noindex on *.pages.dev preview URLs by default; the explicit rule covers custom branch domains.

Step 4: Canonicals to Production

Every page's canonical tag is built from the configured site URL, which is the production URL in every build, so preview pages already pointed their canonicals at production. One template had built canonicals from Astro.url — the request URL — which made each preview canonical to itself. Fixing that template to use Astro.site removed the self-canonical previews, as in Canonical URLs and Duplicate Content on Static Sites.

Step 5: Do Not Block Crawling

A colleague's first instinct was to add Disallow: / to robots.txt on previews. That would have stopped crawlers fetching the pages and seeing the noindex header, leaving already-indexed URLs in the index indefinitely. Previews stay crawlable with noindex until they have dropped out.

Step 6: Remove What Is Already Indexed

With noindex in place, indexed previews drop out as search engines recrawl them, which takes weeks for rarely visited URLs. For the 47 pages where a preview had displaced production, faster action was worth it. Vercel preview domains on vercel.app cannot be verified in Search Console, but the project's branch aliases on a custom preview domain (preview.acme.dev) could; their URLs were submitted to the removals tool, and the production pages were submitted for reindexing. Old Netlify and Pages deploys were deleted where the platform allowed, which turns them into 404s and removes them on the next crawl.

Other Copies Worth Checking

Previews were not the only duplicate copies of the site. The same audit found two more: the origin bucket's direct website endpoint, which served every page without the CDN in front and had been linked from an old status page, and an IP-address URL from a load balancer health check that a crawler had found. Both got the same treatment — noindex headers where the host allowed, and otherwise closing direct access so only the production hostname serves the site. Any hostname that serves the site's HTML other than the canonical one is a potential duplicate; a monthly site: search for the site's distinctive title text, without a domain filter, is a cheap way to find new ones.

Telling Reviewers What Changed

The deploy bot's comment on each pull request now says that the preview is not indexable and should not be shared publicly, with a link to the production URL of the same page. Reviewers who need to share a preview outside the team, for example with a customer, use a password-protected preview instead, which is covered in Password-Protecting Preview Deployments. Since the change, preview URLs appearing in public issues have dropped, though the noindex header means it no longer matters for search when they do.

Step 7: Test It on Every Preview

A check in the preview workflow requests three URLs on the preview and fails unless each has X-Robots-Tag containing noindex; a matching check on production fails if it does. The production check matters as much: a host rule with a too-broad pattern could noindex the real site, which would be a far worse incident.

curl -sI "$PREVIEW_URL/" | grep -qi '^x-robots-tag:.*noindex' || { echo "preview is indexable"; exit 1; }
curl -sI "https://docs.example.com/" | grep -qi '^x-robots-tag:.*noindex' && { echo "PRODUCTION IS NOINDEX"; exit 1; } || true

Measured Impact

Pitfalls & Rollback

• Disallow instead of noindex. It hides the noindex and keeps indexed URLs indexed.
• Relying only on platform defaults. Branch aliases and custom preview domains may not get them.
• Canonicals from the request URL. Use the configured production URL.
• Host rules that also match production. Test production for the absence of noindex.
• Rollback: header rules are scoped by host or context; removing them affects only previews.

Conclusion

Preview deploys leak into search because they are complete public copies of the site linked from public places. Sending X-Robots-Tag: noindex from every non-production host with explicit rules per platform, keeping canonicals on the production URL, leaving previews crawlable so the directive is seen, and testing both previews and production on every deploy removed 311 of 313 indexed preview URLs in twelve weeks and gave 47 production pages their search results back.

FAQ

How do preview deploys end up in search results?

Search engines discover preview URLs through links: a preview URL pasted in a public issue or pull request, shared in a forum, posted in a public chat, or linked from analytics referrer spam. If the preview serves the same pages as production without noindex, it can be indexed and sometimes ranks instead of the real site, splitting signals between duplicate copies.

Is robots.txt Disallow enough to keep previews out?

No. Disallow stops crawling, but a disallowed URL can still be indexed from links alone, shown without a description. It also prevents crawlers from seeing a noindex directive on the page. Use noindex through an X-Robots-Tag header or meta tag, and leave the pages crawlable so search engines can see it.

How do I remove preview URLs that are already indexed?

Add noindex to the preview host so recrawls drop the pages, and for urgent cases verify the preview domain in Search Console and use the removals tool for a temporary removal while the noindex takes effect. For platform subdomains you cannot verify, noindex and time are the only options, which is why prevention matters.

Should previews have canonical tags pointing at production?

It helps. A canonical to the production URL tells search engines which copy is authoritative even before noindex is processed. Generators that build canonicals from a configured site URL do this automatically, as long as the site URL is the production URL in preview builds.

Related

• Parent: Preview Environments for Pull Requests — previews and how to run them safely.
• robots.txt and noindex for Preview and Staging — the directives in depth.
• Password-Protecting Preview Deployments — the stronger option for sensitive previews.
• Cleaning Up Stale Preview Deployments — removing old previews entirely.
• Canonical URLs and Duplicate Content on Static Sites — canonical tags done right.