Keeping Preview Deployments Out of the Index

Every pull request on a modern hosting platform produces a complete, publicly reachable copy of the site, and nobody decides what its indexing policy should be.

When to Use This Approach

Harden preview deployments when:

  • Your platform generates a public URL per branch or per pull request, which is the default on most hosts.
  • Preview links are shared into issue trackers, chat, or client review tools outside your network.
  • A search for your brand plus a platform domain returns anything at all.

Prevention and Remediation Are Different Problems

If nothing has leaked, you need isolation. If something has, you need isolation and removal — and doing them in the wrong order makes the removal impossible.

Prevention path and remediation path A check for already-indexed preview URLs branching into a prevention-only path and a remediation path that adds authentication before requesting removal. anything indexed? no — prevention only isolate and authenticate yes — remediate authenticate first then request removal verify the host first monitor scheduled anonymous fetch Removal before authentication just deletes a copy that will be re-crawled

Implementation Steps

Step 1: Check what has already leaked

# Platform preview domains, common patterns
for d in vercel.app netlify.app pages.dev; do
  echo "== $d"
  curl -s "https://www.google.com/search?q=site:$d+%22Your+Brand%22" \
    -A 'Mozilla/5.0' | grep -oE 'https://[a-z0-9-]+\.'"$d"'[^"]*' | sort -u | head
done

Validation: A more reliable route is Search Console: verify the preview hostname as a property and read its coverage report directly. If any URL reports an indexed state, you are on the remediation path.

Step 2: Detect the environment from the request host

Build-time environment variables are wrong exactly when it matters — a preview built with the production configuration behaves as production.

// lib/environment.js — one function, used everywhere
const PREVIEW_HOST_PATTERNS = [
  /^preview\./,
  /^staging\./,
  /^deploy-preview-\d+--/,
  /\.vercel\.app$/,
  /\.netlify\.app$/,
  /\.pages\.dev$/,
];

export function isPreviewHost(host = '') {
  return PREVIEW_HOST_PATTERNS.some((p) => p.test(host));
}

SEO impact: A host that matches nothing is treated as production, so an unrecognised environment fails in the safe direction — it stays indexable rather than silently suppressing your live site.

Validation: Unit-test the function against your real production hostnames as well as the preview ones. The production assertions are the ones that protect you.

Step 3: Return an authentication challenge

// middleware.js
import { NextResponse } from 'next/server';
import { isPreviewHost } from '@/lib/environment';

export const config = { matcher: '/((?!_next/static|favicon.ico).*)' };

export function middleware(request) {
  const host = request.headers.get('host') ?? '';
  if (!isPreviewHost(host)) return NextResponse.next();

  const expected = 'Basic ' + btoa(`preview:${process.env.PREVIEW_PASSWORD}`);
  if (request.headers.get('authorization') !== expected) {
    return new Response('Preview access requires authentication.', {
      status: 401,
      headers: {
        'WWW-Authenticate': 'Basic realm="preview"',
        'X-Robots-Tag': 'noindex, nofollow, noarchive',
        'Cache-Control': 'private, no-store',
      },
    });
  }

  const res = NextResponse.next();
  res.headers.set('X-Robots-Tag', 'noindex, nofollow, noarchive');
  return res;
}

SEO impact: A crawler that discovers the URL from an external link receives a challenge with nothing to index. The X-Robots-Tag on the authenticated path is the second line of defence, not the mechanism.

Validation: curl -s -o /dev/null -w '%{http_code}\n' https://preview.example.com/ returns 401.

Step 4: Suppress the discovery surfaces

A preview that emits a sitemap is advertising itself, and a preview whose canonical points at itself is asking to be indexed.

// app/sitemap.js
import { headers } from 'next/headers';
import { isPreviewHost } from '@/lib/environment';

export default async function sitemap() {
  const host = (await headers()).get('host') ?? '';
  if (isPreviewHost(host)) return [];        // no URLs advertised from a preview
  return buildProductionSitemap();
}
// The canonical always points at the production origin, never at the current host
export const CANONICAL_ORIGIN = 'https://example.com';

SEO impact: Even if authentication is somehow bypassed, the preview offers no discovery path and claims no canonical of its own.

Validation: curl -s -u preview:$PW https://preview.example.com/sitemap.xml returns an empty URL set, and any page’s canonical points at the production origin.

Step 5: Verify from outside and keep verifying

#!/usr/bin/env bash
# ci/preview-isolation-check.sh — runs on a schedule, not just at deploy
set -euo pipefail
fail=0
for host in preview.example.com staging.example.com; do
  code=$(curl -s -o /dev/null -w '%{http_code}' \
    -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
    "https://$host/")
  if [ "$code" != "401" ]; then
    echo "PREVIEW EXPOSED: $host returned $code"
    fail=1
  fi
done
exit $fail

Validation: Run it from outside your corporate network — a check that passes only because your office IP is allowlisted proves nothing about what a crawler sees.

What Each Control Actually Prevents

The controls are not alternatives; each closes a different failure, and only the combination is complete.

Which control closes which failure A grid of four controls against three failure modes — content readable, URL indexed, and duplicate outranks production — marking which failures each control addresses. content readable URL indexed outranks production noindex header — closed closed cross-host canonical — — partly empty sitemap — partly — 401 challenge closed closed closed Only the challenge closes all three — the rest are defence in depth

SEO Impact Summary

The controls only work together, and the order they are applied in determines whether an existing leak can be cleaned up at all.

Authenticate first, then remove Adding authentication before requesting removal, with the reverse order producing a URL that returns after the removal expires. correct authenticate the host request removal permanent reversed request removal still readable returns in six months
Signal What improves What breaks if misconfigured
Duplicate control The unreleased copy of the site stops competing for canonical selection A host pattern matching production suppresses the live site entirely
Confidentiality Unreleased content, pricing, and campaigns stay unpublished An allowlist based on office IP passes internally and fails for real crawlers
Index hygiene Platform domains stop appearing in brand searches Removing indexed previews without authentication just delays their return
Diagnostics Verified preview properties make leaks visible early An unverified preview host leaks silently until someone searches for it

Measurable signals to watch:

  • HTTP status returned to an anonymous crawler-agent request on each preview host, checked on a schedule.
  • Indexed URL count for the verified preview property, which should be zero.
  • Brand-plus-platform-domain searches, checked after any change to hosting configuration.

Edge Cases and Gotchas

Platform-level password protection may not cover every path Some hosts exempt static assets or serverless function routes from their built-in protection. Test a few asset URLs and an API route directly, not just the homepage.

Basic authentication breaks automated Lighthouse and link checks Your own CI needs credentials too. Pass them explicitly in the checking tools rather than exempting a path, which would create precisely the hole you are closing.

Open graph unfurling is a crawl vector Pasting a preview link into a chat tool causes that tool to fetch the URL. With authentication in place the unfurl fails, which is mildly inconvenient and exactly correct.

Deleting a branch does not remove its indexed URL The deployment disappears and the URL starts returning 404, which eventually drops it — but slowly, and it may be served from cache in the meantime. Authentication prevents the problem; branch hygiene does not solve it. The suppression mechanics are covered in noindex vs robots Disallow for headless routes.

Frequently Asked Questions

Can I just point the preview canonical at production? It helps but does not solve it. A cross-host canonical is a hint that Google can ignore when the pages differ — and they do differ, because the preview holds the unreleased changes. It also leaves the content publicly readable, which is usually the more serious issue.

How did Google find my preview URL in the first place? Almost never by crawling production. The usual routes are a link in a public issue tracker, a chat integration fetching the URL to build an unfurl, or a monitoring tool that exposes it. None of those are affected by your robots.txt.

What do I do about preview URLs that are already indexed? Add authentication so they stop being readable, verify the preview hostname in Search Console, and use the removals tool for immediate suppression. Keep the authentication — the removal expires in about six months, and the challenge is what makes it permanent.

Should preview deployments have a robots.txt that disallows everything? Yes, as a cheap extra layer, but understand it is the weakest of the four controls. It only helps with crawlers that fetch it and honour it, and it does nothing about the content being readable by anyone with the link.

Part of: Preview & Draft Content SEO

Related