Recover CDN edge failures
without guessing.

Edge returns HTTP 200 with a manifest whose MEDIA-SEQUENCE / PDT is from minutes ago. Player keeps the same dead media sequence even though new segments have landed.

Stale-while-revalidate / stale-if-error is wider than the live window budget. Edge served the cached body because origin failed revalidation after a brief blip, and the cache-control directive instructed the edge to keep serving the stale body for hours.

Tighten stale-if-error to under one live window; pin Cache-Control: no-cache, must-revalidate on master playlists; have the agent's cache.freshness probe fail when age > declared max-age.

A path that should be no-store (token URLs, signed segments) is being cached for 60 seconds at the edge — the next viewer hits a 200 with the prior user's signed URL.

A debug or staging endpoint set Cache-Control: max-age=60 and the CDN's ruleset globbed it onto production URLs. The edge applies the header because the rule matched the path, not the contract the origin intended.

Make Cache-Control contracts explicit per content class — origins should always emit no-store on signed URLs and max-age only on unsigned assets; the headers.cache_control_inheritance probe should flag any origin Cache-Control leaked onto a path the manifest treats as private.

Operator fires a soft-purge, the purge API returns 200 OK, and a follow-up curl against the same URL still returns X-Cache: HIT with the old body. Viewers see stale content after a "successful" purge.

The purge API is asynchronous across edge POPs — 200 means the purge was accepted, not propagated. Some POPs received the purge instruction late, or the purge was scoped to a tag the edge didn't fully honor.

Probe edge.purge_acknowledged for at least 30 seconds after a purge; spread the verification curl across multiple geo POPs; require the operator to wait for fan-out ack before considering the purge done.

Cache hit rate drops to single digits on the manifest path despite a clearly warm window. CDN metrics show millions of cache keys for the same URL.

Vary: Accept-Encoding is being honored at edge AND at a parent POP, producing keys the parent POP treats as distinct. A single Accept-Language variant doubles the population, a user-agent tweak triples it.

Collapse Vary keys at the edge before fan-out; only forward Vary to a parent POP if the manifest genuinely varies on that header. headers.vary_collapse should report distinct keys < 4 within minutes of probe start.

EMEA viewers start seeing X-Served-By: cache-jfk8842 — a North America POP. Latency spikes, manifest deliverability drops, and EMEA's health_score degrades within minutes.

GeoDNS / anycast routing table was updated by a deploy that didn't fully sync the POP weights. The new table picks the wrong POP for a region's IP block, and anycast falls back to a less-loaded neighbor.

Pin geo_routing tests to expected POPs; flag any X-Served-By outside the expected region; the edge.geo_drift probe should run as a synthetic check from at least three geo probes against the same URL.

Cache hit rate flatlines near zero. Origin logs spike. Player first-segment latency balloons because every segment fetch is a cold miss.

A query-string cache buster was added at the player layer (a timestamp or a session key), and the CDN is treating every /manifest.m3u8?... URL as a distinct cache key. The cache is empty because nothing maps to it twice.

Strip the busting query string at the edge; have the CDN's cache key only on the canonical path and the variant segments; the cache.availability probe should report a non-zero hit rate within the first probe interval.

Origin connections spike. Origin CPU saturates. Edge metrics show requests bypassing the origin shield and reaching origin directly — yet the shield isn't reporting a fault.

Origin shield reroute rule was configured to bypass on shield latency > 50ms. Under load, the shield crossed 50ms and the bypass rule streered traffic straight to origin, escalating the load.

Tune the reroute threshold against real shield capacity (not a static ms limit); the origin.failover_timing probe must fire when bypass rate spikes without a corresponding shield fault.

Multi-CDN routing pulls traffic to two providers; one is healthy, one is degraded (5xx > 5% for 5 minutes). DNS still weights 50/50, so half the requests go to the unhealthy provider.

DNS weight drift — the multi-CDN controller is slow to converge on the unhealthy provider's health score. While it converges, both providers serve traffic; the degraded provider drags aggregate health down.

Have the multicdn.health_score probe feed DNS weights directly; require controllers to converge within 30 seconds; pin the unhealthy provider to 0 weight until three consecutive probe intervals report health > 0.95.

Inner response header Age: 0 reads cleanly, but the parent metrics show the same URL as a fresh miss every refresh. Player perceives cache misses despite manifest being served from origin only seconds ago.

A redirect chain (origin → parent POP → edge POP) does not propagate the Age header — the inner Age is reset at each hop. The header contract in place ignores Age on the inner response and computes its own age.

Preserve Age across hops; document the Age behavior so monitors can distinguish "edge cache miss" from "redirect reset Age"; if you can't preserve it, surface a synthetic X-Cache-Age header that operators can grep.

Operator runs a purge, the response shows 202 Accepted, and PagerDuty is closed because "purge succeeded". 30 minutes later, viewers report stale content; the edge has not yet purged.

The purge API uses 202 for asynchronous acceptance, but the on-call runbook treats 2xx as success. The actual purge runs asynchronously across the edge fleet; the runbook never waits for fan-out ack.

Treat 202 as "accepted, not done"; have an automated fan-out verification ping 30 seconds post-purge; the edge.purge_acknowledged probe should be part of the on-call runbook before closing the ticket.