Recover failing HLS sessions
without guessing.

Players joining mid-stream sit on a spinner for 8–12 seconds before the first segment lands; tail-end viewers reach the live edge eventually, but the cohort joining at peak pays the cost.

Master playlist TTL is tuned for steady-state refresh, not join load; CDN edge nodes have to revalidate a cold variant window on every fresh viewer. playlist.reachable fires green, but the variant.consistency probe races against CDN edge cache fill.

Pin the master playlist to Cache-Control: no-cache, must-revalidate and prime the variant window with a dedicated low-TTL entrypoint. With Streamwake, the agent reads playlist.reachable + variant.consistency on every cadence and pre-warms the variant tree when it sees a join-rate spike.

Viewers report short rebuffer loops every few minutes, though segment.tail_window is passing and the manifest is current. ABR keeps the chosen rung; nothing changes in the timeline.

Origin egress is saturating during traffic peaks — TTFB drifts from a ~300ms baseline into the 1–2s range. The fetch layer is healthy enough that segment bytes arrive, just late enough to empty the player’s buffer. fetch.ttfb_drift flips to warn before any other probe does.

Throttle the probe fan-out during TTFB excursions (fetch.concurrency auto-throttles when it sees the pool near its ceiling), and route the cohort to a healthier egress. Long term: tier the upstream so the live edge isn’t sharing a fleet with VOD.

Player closes and reopens the SourceBuffer at every SCTE-35 cue-out; the timeline jumps and ABR locks the lowest rung for the rest of the break. Viewers see a hard pause even though the segment returns 200.

The packager emits #EXT-X-DISCONTINUITY at every cue even though PTS is continuous across the splice. MSE treats each tag as a forced reset and burns buffer budget. media.discontinuity_resolution flags a discontinuity at every segment boundary, not just at the actual ad boundary.

Emit #EXT-X-DISCONTINUITY only at genuine PTS, codec, or format transitions, and pair it with the SCTE-35 cue so the agent can correlate them. Have the agent tail-window probe assert a discontinuity_tag:scte35_pair ratio within tolerance.

Tail-window probes pass but viewers report short rebuffers near the live edge. Player log shows the next segment always landing 200–400ms later than the schedule predicted.

Declared #EXTINF:6.0 but the packager is producing 6.21s segments because of GOP boundary alignment — the player schedules the next fetch 200ms too early and stalls before the segment is on the wire. segment.tail_window reports the bytes are reachable; it can’t tell the clock is wrong.

Either coerce the packager to honor the declared target duration, or surface actual segment durations in EXTINF so the player uses the real figure. The agent tail-window probe can flag an EXTINF:declared vs EXTINF:actual drift across the trailing window.

Player keeps the dead MEDIA-SEQUENCE even though new segments have landed on origin; ABR locks at the bottom rung. Logs show the master returning 304-not-modified on every refresh.

CDN honored If-Modified-Since against the prior master timestamp even though the live window advanced — a stale 304 hits the player, the variant playlist never advances, and the cohort stays pinned to the lowest rung until a force-refresh.

Set Cache-Control: no-cache, must-revalidate on the master; pin the variant playlists to max-age=1. The playlist.reachable probe flags a 304 against a moving window as a stale-manifest warning before the player’s ABR notices.

Players pinned to a running session suddenly hit a "MEDIA_SEQUENCE_RESET" error or a hard pause; new joiners are fine. The chunklist appears to step backwards after a brief gap.

The packager restarted and rewrote MEDIA-SEQUENCE without an #EXT-X-DISCONTINUITY tag, so running players fetch a segment URI the packager no longer has. segment.hash_drift catches it because the same logical segment has a different SHA across the rotation.

Either emit #EXT-X-DISCONTINUITY any time MEDIA-SEQUENCE resets (this is what the spec says), or keep the sequence monotonic across the restart by draining the prior packager first. The agent tail-window probe asserts MEDIA-SEQUENCE monotonicity across the last 6 segments.

A low-latency HLS workflow lands the partial segments on time, but the player never commits them — latency sits at 8–12s instead of the expected 2–3s. Reports of a "stuck cursor" at peak.

PART-HOLD-BACK drifted away from the delta cursor after a partial-segment rewrite, or CAN-PRELOAD-BEFORE went out of sync with PART-INF timing. The publish path pushed partials at a different cadence than the playlist claims. llhls.part_holdback_drift catches the mismatch.

Pin partial-segment publication to the same clock that drives the chunklist; assert PART-HOLD-BACK matches the delta cursor on every probe cycle. The agent replays the last 3 partials when the cursor looks stuck, which is usually enough to clear the player.

A fraction of viewers — concentrated in one region — see HTTP 403 on the variant playlist; the others are fine. The cohort rolls over to multi-CDN to clear the issue but the underlying signing mismatch keeps recurring.

The signed-URL region key on the alternate CDN is rotated against a different audience than the player’s GeoIP resolves to; the alternate CDN’s WAF rejects the request before the signed query is even validated. fetch.concurrency and playlist.reachable both report a 403 spike against the new edge.

Audit the multi-CDN signing rules so each region’s signing key matches the GeoIP the player claims; have the agent threshold 4xx rates per BaseURL host so consistent 403s trigger a rollback rather than a flapping roll.

Player picks a variant, MSE throws SourceBuffer.appendBuffer with a QuotaExceededError on the first rendition switch; the player falls back to the original ladder rung.

One variant packages CMAF with ftyp brands isom + dash, another with brands isom + cmfv with no overlap. The shared brand set is empty at the joint, so MSE rejects the next segment on switch. The cmaf.ftyp_match probe catches it as soon as the second variant lands.

Align ftyp brand set across every variant (isom + dash + cmfc is the safe union); update the variant TYPE parameter consistently. Probe flips to pass once the brands overlap.

A subset of fetches returns the same URI but a different SHA-256 byte stream than the same URI served from the primary CDN. Viewers on the rolled-out CDN see slightly stale frames on rendition handoff.

A CDN rotation brought up a stale cache layer that’s still serving the pre-rotation byte stream. The headers match (200, content-length, content-type), so a header-level probe reports green; only a byte-level fingerprint catches it. segment.hash_drift fires.

Pin the new CDN to a fresh cache tier (cold pull) and replay from origin. Have the agent re-hash a leading segment from each CDN per probe cycle and alert on fingerprint mismatch before viewer cohorts notice.

A cohort on the affected tier sits calm on a marquee live broadcast — cohort.reconnect_storm_ratio reads at the pre-event baseline — but the origin-shield tier's queue_depth_concurrent pegs to the slot ceiling while edge.egress_kbps stays flat at the expected rate. The cohort is calm; the upstream origin-shield is the driver.

A correlated upstream cache-miss storm — coincident cache.eviction_posture across the warm edge POPs — fans the replay-origin origin_fetch_qps past the tier's pre-provisioned ceiling. cdn_shield.origin_segment_cache_miss_fanin lifts from its pre-event baseline; segment-leg cache-hit on the affected tier shows cache-miss posture on the warm edge POPs but GREEN at the warm cohort's ladder. The cohort is CALM (cohort.reconnect_storm_ratio at baseline, edge.egress_kbps flat).

Shed the correlated cache-misses to a warm edge POP via origin failover; surface the fail-open posture on the shield circuit-breaker to operator-team for approval. See the working postmortem at /incident-lab/origin-shield-saturation-correlated-cache-miss-storm — Tier 0 (who can shed traffic), Tier 1 (who can fail open), Tier 2 (who can escalate), recovery verified cohort-side + shield-side across four staged gates T+30 s → T+15 m, NOT infrastructure-green.