Measuring Web Vitals in Playwright Tests

Lighthouse measures what happens when a page loads and nobody touches it. Many performance regressions on static sites happen after that: a search modal that takes 400 ms to respond, a filter that freezes on the third keystroke, a "show more" button that shifts the layout, a table of contents that jumps while scrolling. Playwright tests already exercise those flows. Measuring web vitals inside them — LCP and CLS on load, CLS during scrolling, and event timing for each interaction — catches the regressions Lighthouse structurally cannot.

This guide adds vitals measurement to the existing Playwright suite of an Eleventy documentation site, with throttling, thresholds based on measured variance and results posted on each pull request. It belongs to Performance Budgets and Lighthouse CI; the base suite is described in End-to-End Tests for Static Sites with Playwright.

Prerequisites

• A Playwright suite running against a built site in CI (here, npx serve _site in the job).
• Chromium in the Playwright projects; the measurement APIs used are Chromium's.
• @playwright/test 1.45 or later.

Step 1: A Measurement Helper

One helper installs observers before the page loads and reads the results later:

// tests/vitals.ts
import type { Page } from '@playwright/test';

export async function startVitals(page: Page) {
  await page.addInitScript(() => {
    const v = ((window as any).__vitals = { lcp: 0, cls: 0, shifts: [] as number[], events: [] as any[] });
    new PerformanceObserver((l) => { for (const e of l.getEntries()) v.lcp = e.startTime; })
      .observe({ type: 'largest-contentful-paint', buffered: true });
    new PerformanceObserver((l) => { for (const e of l.getEntries() as any[]) if (!e.hadRecentInput) { v.cls += e.value; v.shifts.push(e.value); } })
      .observe({ type: 'layout-shift', buffered: true });
    new PerformanceObserver((l) => { for (const e of l.getEntries() as any[]) if (e.interactionId) v.events.push({ name: e.name, duration: e.duration, target: e.target?.id || e.target?.className }); })
      .observe({ type: 'event', durationThreshold: 16, buffered: true } as any);
  });
}

export const readVitals = (page: Page) => page.evaluate(() => (window as any).__vitals);

addInitScript runs before any page script, so nothing is missed. The CLS value here is a simple sum rather than the session-window maximum that the web-vitals library reports; for a single test flow lasting a few seconds the two are nearly always equal, and the sum is stricter.

Step 2: Throttle the CPU

Developer laptops and CI runners are much faster than readers' phones. Chromium's CDP throttling slows the page's CPU by a fixed factor:

import { test as base } from '@playwright/test';
export const test = base.extend({
  page: async ({ page }, use) => {
    const cdp = await page.context().newCDPSession(page);
    await cdp.send('Emulation.setCPUThrottlingRate', { rate: 4 });
    await use(page);
  },
});

A 4× slowdown on the CI runner approximated a mid-range Android phone closely enough that lab interaction times were within about 30% of field INP for the same widgets — good enough for catching regressions.

Step 3: Measure Real Flows

Three flows that Lighthouse cannot measure:

test('search responds quickly', async ({ page }) => {
  await startVitals(page);
  await page.goto('/docs/');
  await page.keyboard.press('Control+K');
  await page.getByRole('searchbox').pressSequentially('cache', { delay: 80 });
  await page.getByRole('option').first().waitFor();
  const v = await readVitals(page);
  const worst = Math.max(...v.events.map((e) => e.duration));
  test.info().annotations.push({ type: 'perf', description: `worst event ${worst} ms, CLS ${v.cls.toFixed(3)}` });
  expect(worst).toBeLessThan(200);
  expect(v.cls).toBeLessThan(0.02);
});

test('scrolling a long docs page does not shift layout', async ({ page }) => {
  await startVitals(page);
  await page.goto('/docs/configuration/');
  for (let y = 0; y < 8000; y += 500) { await page.mouse.wheel(0, 500); await page.waitForTimeout(100); }
  expect((await readVitals(page)).cls).toBeLessThan(0.02);
});

The third flow — expanding the compatibility table's filters — follows the same pattern. Each test was an existing functional test with three lines added.

Step 4: Set Thresholds From Variance

Before enforcing anything, each test ran 20 times on main in CI to see its spread:

Thresholds sit well above p95 so normal noise never fails a build, and well below where a reader would notice. CLS is essentially deterministic, so its threshold catches any real shift.

Step 5: Report Numbers on Pull Requests

Pass/fail is not enough to see drift. The test annotations are collected by a small reporter that writes a JSON file of every perf measurement, and a CI step compares it with the latest main run and posts a table as a pull request comment, highlighting any measure that moved by more than 25%. Developers see "search worst event 72 → 141 ms" long before it reaches the 200 ms threshold.

What It Caught

In the first three months:

• A search index change that made the first query 310 ms on throttled CPU — the test failed, and the index was split as in Search Index Size Budgets for Large Docs.
• A sticky header redesign that caused CLS 0.06 while scrolling — invisible to Lighthouse, caught by the scroll test.
• A table filter regression from 118 to 230 ms after a new column was added — flagged in the PR comment before reaching the threshold.

How It Fits With the Other Checks

Each check in the pipeline covers a different slice of performance, and the Playwright measurements fill the gap between the page-load lab tools and field data.

The practical rule the team adopted: if a performance problem is reported from the field and no lab check would have caught it, a Playwright flow reproducing it is added in the same pull request as the fix. Over three months that rule added four flows, each a few lines on top of an existing functional test.

Running Against Preview Deploys

The measurements above run against the built site served locally in CI, which keeps them fast and independent of network conditions. A second, smaller run executes the same three flows against the preview deploy URL with network throttling (Network.emulateNetworkConditions over CDP, 1.6 Mbps down and 150 ms latency) to include CDN and header behaviour. It is reported but never fails the build, because preview infrastructure adds noise that local runs avoid; its value is in spotting problems like an uncached search index that only appear through the real host.

Measured Impact

The one false failure came from a CI runner under heavy load during a GitHub incident; a single retry passed, and the suite now retries perf-tagged tests once before reporting failure.

Pitfalls & Rollback

• Unthrottled measurements. Fast CI CPUs hide interaction problems.
• Thresholds from guesses. Measure variance first; set thresholds above p95.
• Asserting in every browser. These APIs are Chromium's; run perf assertions there only.
• Treating lab numbers as field numbers. Use them for regressions; use RUM for truth.
• Rollback: the helper and fixture are opt-in per test; removing the assertions leaves the functional tests intact.

Conclusion

Lighthouse sees the page load; Playwright sees what readers do next. Adding observers, 4× CPU throttling and variance-based thresholds to three existing Playwright flows measured search, filtering and scrolling, caught three regressions Lighthouse CI could not see, and added under a minute to CI.

FAQ

Can Playwright measure Core Web Vitals?

It can measure the browser APIs they are built on. LCP comes from largest-contentful-paint performance entries, CLS from layout-shift entries, and interaction latency from event timing entries, all readable with a PerformanceObserver in page.evaluate. The values are lab measurements, so they track direction and regressions rather than matching field data.

Why measure vitals in Playwright when Lighthouse exists?

Because Playwright can interact. Lighthouse measures a page load with no input, so it cannot measure how long a filter, menu or search takes to respond, and it cannot measure layout shifts that happen while scrolling. Playwright tests already click and scroll through flows, so adding measurement to them covers what Lighthouse misses.

How do I make Playwright performance numbers stable?

Use CPU throttling through the Chrome DevTools Protocol, a fixed viewport, a warm-up run, several runs with the median, and thresholds set from measured variance rather than guesses. Byte counts and layout shift scores are much more stable than timings, so prefer them as hard failures.

Does this work in Firefox and WebKit?

Partly. Layout shift and LCP entries are Chromium APIs, and event timing support varies. CPU throttling through CDP is Chromium only. Run the performance assertions in the Chromium project and keep functional tests in all browsers.

Related

• Parent: Performance Budgets and Lighthouse CI — budgets and their enforcement.
• Lighthouse CI vs WebPageTest for Regression Gates — where page-load gates stop.
• Measuring INP on Static Sites with RUM — the field counterpart.
• Testing Search and Navigation Flows With Playwright — the flows measured here.
• Fixing Poor INP on Static Pages With Heavy Widgets — fixing what these tests find.