Recovering From a Deploy That Deleted Live Assets

At 14:02 a routine deploy finished. By 14:09, the error tracker showed a spike of ChunkLoadError and failed stylesheet loads, and two support messages arrived: "your docs look broken — no styling, menus don't work." The pages being served were fine. The problem was readers who already had pages: their HTML, cached at the CDN and open in browser tabs, referenced CSS and JavaScript files from the previous build — and the deploy had deleted them.

This is a write-up of that incident on a documentation site deployed to S3 behind CloudFront with aws s3 sync --delete: how it was detected, how it was recovered in 11 minutes, and the changes that make it impossible to repeat. It belongs to Rollbacks and Deploy Safety for Static Sites; the preventive pattern is covered in Keeping Old Assets Alive After Deploy.

What Happened

The site's HTML was cached at CloudFront with s-maxage=3600, and invalidated after each deploy — but the invalidation ran after the sync, and took 90 seconds to propagate. Meanwhile, and for HTML pages not in the invalidation list (only changed pages were invalidated), the CDN kept serving old HTML for up to an hour. That HTML pointed at /_astro/index.4f8c1b.css and friends. aws s3 sync --delete had removed them, because the new build had index.9d2e07.css instead.

This had been true of every deploy for months. It became an incident on this deploy because the build tool's upgrade had changed the hash of every asset at once, so every page with cached HTML broke instead of a few.

Detection

The alert on JavaScript errors fired; there was no alert on asset 404s, which would have fired seven minutes earlier. Real-User Error Tracking on Static Sites covers the tracking that caught it.

Recovery

The on-call engineer's steps, in order:

1. Restore the previous assets (minute 9–13). S3 versioning was enabled on the bucket, so the deleted objects existed as noncurrent versions. A script listed delete markers created by the deploy and removed them, restoring 412 objects:

aws s3api list-object-versions --bucket "$BUCKET" --prefix _astro/ \
  --query 'DeleteMarkers[?IsLatest && LastModified>=`2026-06-10T14:00`].[Key,VersionId]' --output text |
while read key vid; do aws s3api delete-object --bucket "$BUCKET" --key "$key" --version-id "$vid"; done

2. Invalidate all HTML (minute 13–15) with /*.html and /*/ patterns so readers stopped receiving old HTML as soon as possible.
3. Verify (minute 15–20): the CDN's 404 rate for /_astro/ returned to baseline, and the error tracker's ChunkLoadError rate fell to zero.

Total time from alert to resolution: 11 minutes. Without bucket versioning, the fallback would have been checking out the previous commit, rebuilding and uploading its assets — about 6 minutes longer.

Impact

About 3,900 page views over 20 minutes received broken pages — 18% of views in that window. No data was lost; the cost was a degraded experience and support time.

Prevention

Four changes, each addressing a different part of the chain:

1. Never delete hashed assets on deploy. The sync for _astro/ no longer uses --delete. A weekly job deletes hashed assets that are not referenced by any of the last five builds' manifests and are older than 7 days. Storage cost of keeping them: about 40 MB, negligible.

aws s3 sync dist/_astro "s3://$BUCKET/_astro" --cache-control 'public, max-age=31536000, immutable'   # no --delete
aws s3 sync dist "s3://$BUCKET" --exclude '_astro/*' --delete --cache-control 'public, max-age=0, s-maxage=3600'

2. Upload assets before HTML. New assets go up first, then HTML, so there is never a moment where new HTML references assets that have not arrived.

3. Alert on asset 404s. A CloudFront metric alarm on the 4xx rate for /_astro/* fires above 0.5% for two minutes — it would have caught this in two minutes rather than nine.

4. Keep versioning on. Bucket versioning with a 30-day noncurrent-version expiry stays enabled, because it made recovery a script instead of a rebuild.

Moving to atomic deploys with versioned prefixes, described in Rolling Back S3 and CloudFront Deploys With Versioned Prefixes, removes this class of problem entirely and was done the following quarter.

What the Post-Incident Review Changed Beyond the Deploy

The review looked past the deploy script at why the problem had existed unnoticed for months. Three findings led to changes outside the pipeline itself.

First, the deploy script had been copied from a blog post when the site was small and deployed weekly. Nobody had revisited it as the site grew to several deploys a day, which multiplied the number of windows in which cached HTML could outlive its assets. Deploy scripts now have an owner listed in the repository's CODEOWNERS file, and changes to them need that owner's review.

Second, small numbers of asset 404s had shown up in the CDN logs after most deploys for months — 0.1% to 0.3% for a few minutes — and had been dismissed as noise from bots. They were readers with open tabs. The new alarm's threshold was set below that level so the pattern would have been visible.

Third, the runbook for "site looks broken" had started with "check whether the latest deploy succeeded", which it had. The runbook now starts with the asset 404 rate and the error tracker, and includes the version-restore script above, tested in a quarterly drill against a staging bucket. The first drill found that the script's date filter used the wrong time zone, which would have restored nothing — exactly the kind of mistake better found in a drill than during an incident.

Communicating During the Incident

A short status note went on the site's status page at 14:13, two minutes after the alert, saying that some readers might see unstyled pages and that a fix was in progress, and was resolved at 14:24. Support used a canned reply asking affected readers to reload, which fixed it for anyone whose HTML had already been refreshed at the edge.

Measured Impact of the Changes

Pitfalls & Rollback

• --delete on hashed asset folders. Old HTML still references them.
• Uploading HTML before assets. New pages briefly reference missing files.
• Alerting only on JavaScript errors. Alert on asset 404s directly.
• No versioning on the bucket. It turns recovery into a rebuild.
• Rollback: restoring deleted objects from versions is the rollback; the runbook script above does it in minutes.

Conclusion

Deleting the previous build's assets during a deploy breaks every page still served from cached HTML. On this site an asset hash change made it an incident affecting 18% of views for 20 minutes; bucket versioning made recovery an 11-minute script. Keeping old hashed assets for a week, uploading assets before HTML, alerting on asset 404s and later moving to versioned prefixes mean it cannot happen again.

FAQ

Why does deleting old assets on deploy break pages?

HTML is cached for some time in CDNs and browsers, and tabs stay open for hours. That cached HTML references the previous build's hashed CSS and JavaScript files. If the deploy deletes those files, any reader with old HTML requests assets that no longer exist, gets 404s, and sees an unstyled or broken page until their HTML refreshes.

How long should old assets be kept after a deploy?

At least as long as the longest time old HTML can be served or stay open, which on most static sites means the CDN's HTML TTL plus a margin for open tabs. Keeping the previous few builds' assets for seven days covers almost every case at negligible storage cost, because hashed assets are small and change rarely.

How can I recover quickly if assets were deleted?

Redeploy the previous build's assets alongside the current build, or restore them from object storage versioning if it was enabled. Purging HTML from the CDN shortens the window during which readers request old assets. Atomic-deploy hosts avoid the problem because previous deployments stay available.

Do atomic deploy hosts have this problem?

Less so. Hosts such as Netlify, Vercel and Cloudflare Pages keep every deployment's files, and many serve assets from previous deployments for some time. The problem appears mostly with self-managed deploys to object storage or servers using sync tools that delete files missing from the new build.

Related

• Parent: Rollbacks and Deploy Safety for Static Sites — deploying without breaking readers.
• Keeping Old Assets Alive After Deploy — the preventive pattern in depth.
• Atomic Deploys vs Incremental Uploads — why atomic hosts avoid this.
• Cache-Control Headers on S3 and CloudFront — the caching that set the window.
• Post-Deploy Verification Checklists — checks that run after each deploy.