Auditing Content Parity After a Migration

A static site migration usually ends with someone clicking through twenty pages on the preview and declaring it good. Then, over the following months, readers find the rest: a table that lost a column because the new Markdown renderer handles pipes in code differently, a set of pages that lost their images because a shortcode was not ported, a section whose code samples lost syntax highlighting, meta descriptions that fell back to the site default on every blog post. Each is small; together they are why migrations are often followed by a slow decline in search traffic that nobody can explain.

This guide replaces clicking around with a systematic audit: extract a structured summary of every page from the old and new builds, compare them page by page and element by element, and triage differences until only intentional ones remain. It is part of Migrating Between Static Site Generators. The numbers come from migrating a 1,640-page Jekyll documentation site to Hugo, the same project described in Migrating a Docs Site from Jekyll to Hugo.

Prerequisites

• Both builds available: the old site's final build (or the live site) and the new site's preview.
• The URL mapping from old to new, including redirects — see Keeping Redirects Working After an SSG Migration.
• Node 20+ with node-html-parser, or Python with selectolax; a crawler such as Screaming Frog for the pre-launch pass.

Step 1: Define What "Same" Means

Byte-for-byte comparison is useless: every generator produces different markup. Parity has to be defined in terms readers and search engines care about. For each page, extract:

• Identity: URL, HTTP status, canonical, title, meta description, lang.
• Structure: the heading outline (level and text of every h1–h4).
• Substance: word count of the main content, a normalized text fingerprint, and counts of paragraphs, list items, table rows and code blocks.
• References: internal link targets, external link targets, image sources with alt text.
• Machine data: JSON-LD types and key properties, Open Graph image.

The main content has to be isolated from navigation, headers and footers, which differ by design. A CSS selector per site (main article on Jekyll, main .content on Hugo) handles that.

Step 2: Extract Summaries From Both Builds

One script runs over each build and writes a JSON file keyed by old URL. For the new build, each page's URL is mapped back to its old URL through the redirect map, so pages that moved are still compared with their predecessor.

// scripts/parity/extract.mjs <buildDir> <mainSelector> <out.json> [urlMap.json]
for (const file of await globby(`${dir}/**/*.html`)) {
  const doc = parse(await readFile(file, 'utf8'));
  const main = doc.querySelector(mainSelector);
  if (!main) continue;
  const url = toUrl(file, dir);
  const key = urlMap?.[url] ?? url;
  pages[key] = {
    url, title: text(doc.querySelector('title')), desc: attr(doc, 'meta[name=description]', 'content'),
    canonical: attr(doc, 'link[rel=canonical]', 'href'),
    outline: main.querySelectorAll('h1,h2,h3,h4').map((h) => `${h.tagName}:${norm(h.text)}`),
    words: norm(main.text).split(' ').length,
    fingerprint: simhash(norm(main.text)),
    tables: main.querySelectorAll('tr').length, code: main.querySelectorAll('pre').length,
    links: main.querySelectorAll('a[href]').map((a) => resolveInternal(a.getAttribute('href'))).sort(),
    images: main.querySelectorAll('img').map((i) => `${basename(i.getAttribute('src'))}|${i.getAttribute('alt') ?? ''}`).sort()
  };
}

The text fingerprint is a SimHash of normalized words; two pages with a Hamming distance above a threshold have meaningfully different text even when word counts are close. Image comparison uses base file names because paths and optimized formats change legitimately between generators.

Step 3: Compare and Classify

A comparison script walks every old URL and classifies it:

Pages can fall into several classes; the table counts each page's most severe class. Tolerances matter: a word count within 2% and a fingerprint distance below 6 bits counted as identical, which absorbed differences in how the two generators handled smart quotes and footnote markers.

Step 4: Triage by Cause, Not by Page

Four hundred differing pages sounds like weeks of work. In practice, differences cluster by cause, and fixing a cause fixes many pages at once. Grouping differences by the kind of change (for example "image count dropped by exactly the number of {% include figure %} tags") surfaced the causes on this migration:

• Metadata (187 pages): Jekyll's jekyll-seo-tag used excerpt as the description fallback; the Hugo template used the site description. One template fix.
• References (142 pages): a figure include with a caption parameter had been ported without handling images in subdirectories. One shortcode fix.
• Substance (91 pages): Kramdown and Goldmark treat pipes inside inline code in tables differently; 68 tables lost columns. One render setting plus 23 content escapes.
• Structure (64 pages): headings inside <details> blocks written in raw HTML were no longer rendered as Markdown. Goldmark unsafe = true plus a content fix.
• Missing (23 pages): pages with permalink overrides that the frontmatter converter had ignored. Converter fix, rerun.

Five template or converter fixes resolved 470 of the 507 differing pages. The remainder were reviewed by hand.

Step 5: Record Intentional Differences

Not every difference is a bug. Sections were renamed, tag pages were dropped, and some pages were merged deliberately. Each intentional difference goes into an allowlist with a reason:

# parity-allowlist.yaml
- match: '/tags/**'
  reason: tag pages removed; redirected to search
- match: '/guides/legacy-setup/'
  fields: [outline, words]
  reason: merged two legacy pages into one during migration (DOC-412)

The final report shows only unexplained differences. Before launch it was empty, and the allowlist — 19 entries — became the record of what the migration changed on purpose, which was useful when a stakeholder later asked why a page looked different.

Step 6: Crawl Before Launch

File comparison catches rendering differences; a crawl of the preview catches host behaviour — redirects, status codes, trailing slashes, headers. Crawl the old live site and the new preview with the same crawler settings, export URL, status, title, canonical and indexability, and diff. On this project the crawl found 6 issues the file comparison could not: redirects that chained through two hops, and a trailing-slash rule that turned four old .html URLs into 404s instead of redirects.

Measured Impact

The previous migration referenced here was the same team's earlier move of a smaller site, done without an audit; its traffic took five months to recover.

Pitfalls & Rollback

• Comparing whole pages. Template differences drown content differences; compare the main region only.
• Tolerances too tight or too loose. Calibrate on a few pages you know are identical and a few you know are not.
• Fixing page by page. Group by cause first; most differences share one.
• Ignoring new-only pages. Generated pages the old site never had can create thin or duplicate content.
• Rollback: the audit changes nothing on the site; if the report shows a problem after launch, the fix is a normal deploy, and the old build remains the reference.

Conclusion

Content parity is provable. Extract a structured summary of every page from both builds, compare them per URL with sensible tolerances, fix causes rather than pages, and keep an allowlist of intentional changes. On this migration five days of audit work turned 507 differing pages into zero unexplained ones, and the site gained traffic after launch instead of losing it.

FAQ

What does content parity mean in a migration?

That every page readers and search engines could reach on the old site is reachable on the new site, with the same substance, including text, headings, links, images, code samples, metadata and structured data, either at the same URL or through a redirect.

Why not just compare page counts?

Page counts hide both losses and additions. A migration can drop 30 pages and add 30 generated tag pages and show the same total. Parity has to be checked per URL and per element, such as headings, links and images on each page.

Should the comparison use built files or a crawl?

A crawl of each running site catches everything a reader sees, including redirects, host rules and client-rendered content. Comparing built files is faster and good for iteration. On this migration both were used, files during development and crawls before launch.

How do I handle intentional differences?

Record them in an allowlist with a reason, such as a renamed section, removed tag pages or a new footer, so the audit report only shows unexplained differences. The allowlist also becomes documentation of what the migration changed on purpose.

Related

• Parent: Migrating Between Static Site Generators — the full migration process.
• Converting Front Matter at Scale During Migration — where many parity issues start.
• Porting Shortcodes and Includes Between Generators — the usual cause of reference differences.
• Snapshot Testing Build Output — the same idea as an ongoing test.
• Migrating from Docusaurus to Starlight — another migration this audit applies to.