CMAF low-latency edge-cache warm-up
on a primetime LL-HLS lane.
A working postmortem on a weekday primetime LL-HLS / CMAF event in which newly activated CDN edge POPs returned incomplete or delayed partial-segment bodies during cache warm-up. The live HLS manifests on the affected POPs returned clean (200 OK on the playlist, no 5xx), the segment-leg availability was unaffected, but partial-segment fetches on the affected regional cohort ran wider than the cohort's PART-HOLD-BACK window — so the player fallback loop on first join climbed from 0.6% to ~9% of cohort under a 4-minute cache warm-up crescendo before the cache tail settled. The Streamwake agentic ops layer caught it from viewer-impact telemetry, ranked the failure edge_partial_segment_warmup at 84% confidence, and remediated with a partial-segment prewarm + cache-tail-pin rebalance — split explicitly into the acts the agent did autonomously and the acts it surfaced to humans.
Book a technical demo for CMAF low-latency warm-up
Read the postmortem — then bring your own incident to Streamwake.
Two ways to engage on this exact failure pattern: book a 30-minute technical demo where we walk through the probe cascade on your source, or hand us an archived incident and watch the agent diagnose it end-to-end.
Both routes land on the scoping intake form — no SDR gate.
What the cohort saw
The first things to read on any real primetime cache-warm-up incident are the cohort-level numbers — how many sessions the partial-segment fallback hit, how long the cache tail took to settle, and where the symptom landed on the affected edge cohort. Three numbers did the heavy lifting here. The figures below are simulated telemetry — the disclosure above applies to every figure on this page.
Roughly 9% of the na-east primetime cohort — session-level hls.js partial-segment fallback on the affected cdn-A/atl01 edge cohort, where the partial-segment bucket had not yet primed and the cache tail ran wider than the cohort PART-HOLD-BACK profile. The player fell back to fetching the next full segment instead of the next partial in flight, dragging time-to-first-frame and pushing the cohort into the startup-stall lane; cohort_partial_segment_404_ratio climbed 16× above baseline.
~24 minutes between the first partial-segment variance at T+0m and the edge_partial_segment_availability_ratio climbing back to 0.92 at T+22m. The postmortem window goes longer because the playbook pre-peak anchor and the CDN vendor's partial-segment prewarm on activation ship on the next deploy / next release train — not in the same incident window.
segment_partial_age_ms: 1.86 on cdn-A/atl01 (na-east cohort) — newly activated edge POP, partial-segment bucket primed 4 minutes after the segment bucket. The same cohort on the same vendor's warm POP reads 0.42s (baseline); the discriminator is on the warm-up cadence, not on the segment leg.
How Streamwake classified this incident
Three ranked hypotheses, with the top one filing the timeline and the secondary signal carrying the cause. The isp_regional_congestion_on_partial_path and encoder_partial_segment_timing lanes are included to make explicit that the agent ruled them out — the symptom was on the partial-segment cache warm-up leg, not on a regional ISP fault and not on an encoder-side fault that would have failed the warm POP cohort on the same encoder.
edge_partial_segment_warmup — newly activated edge POPs run the cache warm-up that primes the segment bucket only; the partial-segment bucket lags by 30–180s. Three signals line up: edge.edge_partial_segment_availability_ratio 0.41 vs 0.92 baseline on the affected regional cohort, segment_partial_age_ms 1.86s vs 0.42s baseline (exceeds PART-HOLD-BACK),cohort.cohort_partial_segment_404_ratio 0.157 vs 0 baseline over the 90s window.
isp_regional_congestion_on_partial_path was the secondary signal ranked at 7% — the warm POP's edge_partial_segment_availability_ratio matches baseline on the same ASN; the cohort's isp.cell_load.on_affected_region reads pass. The dismissal rule was"rank the cause on the broader probe pattern, not on the single symptom that landed on the player"; the cache-locale nature of the failure is the decisive signal pattern.
- Region: na-east (atl01 edge)
- Status: resolved (window closed)
- Opened: 2026-08-12 19:11 UTC
- Spread: contained to na-east — eu-west and apac unaffected; warm POPs on the same vendor reads at baseline.
Above the 80% threshold the agent treats as a confident top-hypothesis filing. encoder_partial_segment_timing · 0.06 was cleared explicitly because the partial-segment fetch legs fail on the affected POP only — not on the warm POP on the same encoder — so the failure is cache-locale-shaped, not encoder-shaped.
Incident timeline
Ten events: detection on the cohort, classification across three ranked hypotheses, four autonomous acts the agent took on its own, three acts it surfaced to humans, the recovery probe, and the resolution. The right-hand "act" tag is what makes this postmortem distinct from a generic write-up — it pins the split between autonomous agentic ops and the work that genuinely needed a person. All times below are simulated telemetry — the disclosure at the top of this page applies to every minute offset on the timeline.
Today
- T+0mDetectionby cohort agent · na-east (atl01 edge)act · autonomous
Partial-segment availability variance on the affected edge POP only
edge.edge_partial_segment_availability_ratio dropped from 0.92 baseline to 0.41 over a 90-second window on the cdn-A/atl01 cohort; segment_partial_age_ms climbed from 0.42s to 1.86s (exceeds PART-HOLD-BACK of ~1.0s); cdn_pop.warm_up_age_s read 244s and cdn_pop.partial_segment_warmup_state read partial — newly activated edge cohort, partial bucket not yet primed.
Aug 12, 07:11:42 PM - T+1mClassificationby Streamwake reliability agentact · autonomous
Ranked: edge_partial_segment_warmup (0.84) · secondary isp_regional_congestion_on_partial_path (0.07) · encoder_partial_segment_timing (0.06)
Top hypothesis reads 84% confidence. Three ranked alternatives dismissed: isp_regional_congestion_on_partial_path (the warm POP on the same ASN reads pass — discriminator for the cache-locale nature of the failure), encoder_partial_segment_timing (encoder timing reads continuously and the partial-segment fetch legs fail on the affected POP only, not the warm POP on the same encoder — discriminator for the cache-locale shape).
Aug 12, 07:12:42 PM - T+2mAutomated actionby Streamwake reliability agentact · autonomous
Emitted prewarm_partial_segment_bucket_on_activation + pin_partial_segment_cache_tail_to_PART_HOLD_BACK
Pre-warm the partial-segment bucket on the affected edge POP from a clean source window anchored on the cohort PART-HOLD-BACK profile; pin the partial-segment cache tail to PART-HOLD-BACK so the partial-segment layer primes from the cohort cadence, not the segment cadence. The reissued cache tail is what the cohort reads for the rest of the window.
Aug 12, 07:13:42 PM - T+3mAutomated actionby Streamwake reliability agentact · autonomous
Ingest probe: ingest_partial_segment_tail_probe queued
Queue the partial-segment tail probe as a 30-second-cadence cohort sample with a probe arch on the cohort revalidation cadence. The probe gives the audit step a probe-as-source for next time's pre-peak cache warm-up window.
Aug 12, 07:14:42 PM - T+5mAutomated actionby Streamwake reliability agentact · autonomous
Rebalance hint queued: prewarm_partial_segment_bucket_on_activation_in_playbook / pre-peak window
Lift prewarm_partial_segment_bucket_on_activation into the playbook pre-peak script so future newly activated edge POPs run the partial-segment prewarm on activation alongside the segment pre-warm. Staged for the next-day deployment so the operator team can review the false-positive rate.
Aug 12, 07:16:42 PM - T+11mSurfaced to humanby agent → CDN vendoract · surfaced to humans
CDN vendor engaged on partial-segment prewarm on newly activated POPs
Vendor confirmed that the partial-segment bucket warms asynchronously and is meant to lag the segment bucket by 30–180s. New policy: prime the partial bucket on POP activation alongside the segment bucket, anchored on the cohort PART-HOLD-BACK profile. Vendor confirmed the warm-up support ships on the next CDN release train.
Aug 12, 07:22:42 PM - T+14mSurfaced to humanby on-callact · surfaced to humans
On-call paged for the partial-segment warm-up root-cause review
Page acknowledged within 92s; reviewer confirmed the agent's rank and that the prewarm and cache-tail-pin had prevented the cohort from doubling into a player-startup-stall clip before the reissue landed.
Aug 12, 07:25:42 PM - T+18mSurfaced to humanby reliability teamact · surfaced to humans
Postmortem write-up assigned (this page)
Reliability team assigned the public postmortem; this page is the resulting write-up, with the ranked hypotheses, recommended fix, and agentic-act split pinned from the timeline.
Aug 12, 07:29:42 PM - T+22mAutomated actionby Streamwake reliability agentact · autonomous
Re-probed the cohort; edge_partial_segment_availability_ratio returned within baseline
edge.edge_partial_segment_availability_ratio climbed from 0.41 to 0.92 over the next 90 seconds; segment_partial_age_ms cleared to 0.51s; cohort.player_startup_fallback_to_join_ratio dropped from 0.092 to 0.011; cohort_partial_segment_404_ratio settled at 0.023 (below the 0.05 false-positive tolerance floor).
Aug 12, 07:33:42 PM - T+35mResolutionby Operator + agentact · autonomous
Incident resolved; partial-segment prewarm + cache-tail-pin ship next day; vendor warm-up policy uplift ships next release
Player-side hls.js fallback loop cleared for the affected regional cohort; no further partial-segment leg variance on the reissued cache tail; the prewarm playbook anchor and the vendor partial-segment prewarm on activation both ship in the next deploy window / next CDN release train.
Aug 12, 07:46:42 PM
- classify · ranked three hypotheses with confidence in 90s
- prewarm · emitted prewarm_partial_segment_bucket_on_activation on the affected edge cohort
- pin · emitted pin_partial_segment_cache_tail_to_PART_HOLD_BACK on the cohort cadence
- ingest · queued ingest_partial_segment_tail_probe for the audit step
- anchor · lifted prewarm_partial_segment_bucket_on_activation into the playbook pre-peak script
- CDN vendor · engaged on partial-segment prewarm on newly activated edge POPs
- on-call · paged for the partial-segment warm-up root-cause review
- reliability team · assigned the public postmortem write-up (this page)
Anatomy of the evidence packet
The two packets on the failing source — a CMAF low-latency warmup probe packet on the LL-HLS delta update (with the partial-segment availability ratio collapsed, the cache tail wider than PART-HOLD-BACK, and the cohort's partial-segment 404s spiked), and the agent timeline response with the ranked hypotheses and the rebalance hints. The probe packet is what the agent decided on; the timeline response is what the agent emitted.
GET /live/event/stream.m3u8 HTTP/1.1
host: cdn.example.com
accept: application/vnd.apple.mpegurl
----- cycle 0 (T+0m, before partial-segment prewarm + cache-tail-pin) -----
# playlist returned clean by the affected edge POP, partial bodies lagging
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=24
x-cdn: cdn-A/atl01 ← newly activated POP, 4-minute cache tail
x-packager: pkg-prime-04
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-PART-INF:PART-TARGET=0.33
#EXT-X-MEDIA-SEQUENCE:68421
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000,
068421.ts
#EXT-X-PART:DURATION=0.330,URI="068421.0.ts",INDEPENDENT=YES
#EXT-X-PART:DURATION=0.330,URI="068421.1.ts"
#EXT-X-PART:DURATION=0.330,URI="068421.2.ts"
#EXT-X-PART:DURATION=0.330,URI="068421.3.ts"
#EXT-X-PART:DURATION=0.330,URI="068421.4.ts"
# edge_partial_segment_availability_ratio: 0.41 (cohort mean on affected POP)
# segment_partial_age_ms: 1.86 (baseline 0.42s — above PART-HOLD-BACK)
# edge_partial_fetch_404_ratio: 0.18 (partial-segment 404s vs baseline 0)
# playlist_manifest_fetch_ok: pass (chunklist 200 on the same POP)
# segment_full_fetch_ok: pass (segment leg green, no 5xx)
# cohort.player_startup_fallback_to_join_ratio: 0.092 (vs 0.006 baseline over 90s)
# cohort.cohort_hls_join_stall_ratio: 0.108 (vs 0.014 baseline, 90s window)
# cohort.cohort_partial_segment_404_ratio: 0.157 (vs 0 baseline, 90s window)
# encoder.bitrate_target_delta_pct: 0.34 (within baseline — encoder healthy)
# origin.aligned_target_kbps_baseline: 4211 (stable; origin envelope green)
# cdn.cache_origin_hit: pass (segment leg green on the same POP)
# cdn_pop.warm_up_age_s: 244 (newly activated POP — 4-minute cache tail)
# cdn_pop.manifest_concurrency_ceiling_pct: 0.71 (within the 90% pre-peak ceiling)
# cdn_pop.partial_segment_warmup_state: partial (segment bucket primed, partial bucket not yet)
# isp.cell_load.on_affected_region: pass (cell-load 0.42 vs 0.51 baseline reference)
# player.abr_oscillation.cross_warm_pop: pass (Playback ABO ladder reads clean on warm POPs)
----- cycle 1 (T+0m +6s, after partial-segment prewarm + cache-tail-pin) -----
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=22
x-cdn: cdn-A/atl01
x-packager: pkg-prime-04
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-PART-INF:PART-TARGET=0.33
#EXT-X-MEDIA-SEQUENCE:68423
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000,
068423.ts
#EXT-X-PART:DURATION=0.330,URI="068423.0.ts",INDEPENDENT=YES
#EXT-X-PART:DURATION=0.330,URI="068423.1.ts"
#EXT-X-PART:DURATION=0.330,URI="68423.2.ts"
# edge_partial_segment_availability_ratio: 0.92
# segment_partial_age_ms: 0.51 (within baseline / within PART-HOLD-BACK)
# edge_partial_fetch_404_ratio: 0.02 (within false-positive floor)
# cohort.player_startup_fallback_to_join_ratio: 0.011
# cohort.cohort_partial_segment_404_ratio: 0.023 (within tolerance 0.05)
# rebalance_hints emitted: prewarm_partial_segment_bucket_on_activation,
# pin_partial_segment_cache_tail_to_PART_HOLD_BACK,
# ingest_partial_segment_tail_probe- edge_partial_segment_availability_ratio →
0.41 (vs 0.92 baseline on the affected cohort) - segment_partial_age_ms →
1.86 (exceeds PART-HOLD-BACK) - cohort_partial_segment_404_ratio →
0.157 vs 0 baseline (90s window) - player_startup_fallback_to_join_ratio →
0.092 vs 0.006 baseline (90s window) - segment_full_fetch_ok → pass. Segment leg green on the same cohort. playlist_manifest_fetch_ok → pass. Manifest served 200 on the same POP.
{
"stream_id": "cklivewarmupcmafprimetime9117",
"source": "https://cdn.example.com/live/event/stream.m3u8",
"protocol": "LL-HLS / CMAF",
"checked_at": "2026-08-12T19:11:42Z",
"ranked_hypotheses": [
{
"rank": 1,
"hypothesis": "edge_partial_segment_warmup",
"confidence": 0.84,
"evidence_signals": [
"edge.edge_partial_segment_availability_ratio → fail (0.41 vs 0.92 baseline on the affected regional POP cohort)",
"edge.segment_partial_age_ms → fail (1.86s vs 0.42s baseline; exceeds PART-HOLD-BACK of ~1.0s)",
"cohort.cohort_partial_segment_404_ratio → fail (0.157 vs 0 baseline, 90s cohort window)",
"cohort.player_startup_fallback_to_join_ratio → fail (0.092 vs 0.006 baseline over the same 90s window)",
"cohort.cohort_hls_join_stall_ratio → fail (0.108 vs 0.014 baseline)",
"cdn_pop.warm_up_age_s → 244 (newly activated POP — 4-minute cache tail at the partial bucket)",
"cdn_pop.partial_segment_warmup_state → partial (segment bucket primed, partial bucket not yet primed)",
"playlist_manifest_fetch_ok → pass (chunklist 200 on the affected POP, 22–24ms server-timing)",
"segment_full_fetch_ok → pass (segment leg green; the missure is on the partial-segment leg alone)",
"encoder.bitrate_target_delta_pct → pass (within baseline 0.34; encoder envelope healthy)",
"origin.aligned_target_kbps_baseline → pass (4211 kbps, stable; origin envelope green)",
"cdn.cache_origin_hit → pass (segment leg serves from cache on the affected POP)"
]
},
{
"rank": 2,
"hypothesis": "isp_regional_congestion_on_partial_path",
"confidence": 0.07,
"evidence_signals": [
"isp.cell_load.on_affected_region reads pass; cohort geofan shows the affected and warm POP cohorts on the same ASN; if a regional ISP fault were stateful, the warm POP would also misfire"
]
},
{
"rank": 3,
"hypothesis": "encoder_partial_segment_timing",
"confidence": 0.06,
"evidence_signals": [
"encoder emits PART-TARGET on the 0.33s cadence continuously; the partial-segment fetch legs fail on the affected POP cohort only, not on the warm POP cohort on the same encoder — so the failure is cache-locale-shaped, not encoder-shaped"
]
}
],
"agent_rebalance_hints": [
"prewarm_partial_segment_bucket_on_activation",
"pin_partial_segment_cache_tail_to_PART_HOLD_BACK",
"ingest_partial_segment_tail_probe"
],
"surfaced_to_humans": [
{"owner": "CDN vendor", "task": "engage on partial-segment prewarm on newly activated POPs"},
{"owner": "on-call", "task": "page for the partial-segment warm-up root-cause review"},
{"owner": "reliability team", "task": "assign the postmortem write-up (this page)"}
]
}- rebalance hint →
prewarm_partial_segment_bucket_on_activation - rebalance hint →
pin_partial_segment_cache_tail_to_PART_HOLD_BACK - rebalance hint →
ingest_partial_segment_tail_probe - surfaced → engaged the CDN vendor on partial-segment prewarm on newly activated edge POPs
- surfaced → paged on-call for the partial-segment warm-up root-cause review
Detection, classify, mitigate, recover (simulated telemetry)
Four timing windows on the postmortem timeline, each read off the cohort probe cadence. The figures are simulated telemetry — the disclosure near the top of this page applies to every figure on this list.
~5 s
cohort.cohort_partial_segment_404_ratio climbed from 0 to 0.05 within the first 30 s window; the agent surfaced the detector from the cache-warm-up variance at T+5 s.
~1 m
edge_partial_segment_warmup ranked at 0.84 confidence with three ranked hypotheses at T+1 m — discriminator is the warm POP on the same vendor reads pass.
~9 m
prewarm_partial_segment_bucket_on_activation + pin_partial_segment_cache_tail_to_PART_HOLD_BACK landed at T+9 m; the affected cohort re-anchored by T+11 m.
~24 m
edge_partial_segment_availability_ratio returns to 0.92 baseline at T+22 m; cohort_hls_join_stall_ratio clears at T+24 m — playbook + vendor uplift ship next deploy / release train.
Partial-segment prewarm now · cache-tail-pin next time · vendor uplift next release
Three changes close the gap the timeline exposed. The first two are what the agent already emitted on this incident (the partial-segment prewarm + cache-tail-pin); the last is what ships on the next CDN release train. All three are tracked as exported rebalance_hints rows.
Take the partial-segment layer off the affected edge cohort, snap the partial bucket from a clean source window anchored on the cohort's PART-HOLD-BACK profile, and pin the cache tail to the cohort cadence. The reissued cache tail is what the cohort reads for the rest of the window — the partial-segment leg of the affected POP no longer lags the segment leg.
edge.edge_partial_segment_availability_ratio climbs above 0.90 on the affected cohort within ten consecutive 30-second windows;segment_partial_age_ms clears under 0.55 s on the same cadence; cohort.cohort_partial_segment_404_ratio settles under 0.05 (within the cohort's false-positive tolerance).
Currently the pre-peak script primes the segment bucket on newly activated edge POPs only; it does not prime the partial-segment bucket. The bumped pre-peak script primes the partial-segment bucket on activation anchored on the cohort's PART-HOLD-BACK profile, so newly activated edge POPs run the partial-segment prewarm on activation alongside the segment pre-warm. Staged for the next deploy so the operator team can review the false-positive rate.
cdn_pop.partial_segment_warmup_state stays at primed on every newly activated edge POP across a primetime cycle;cdn_pop.warm_up_age_s on the partial bucket stays under 60 s; the audit-loop probe ingest_partial_segment_tail_probe sends its first cohort probe within the next 24 h.
Want Streamwake to catch this on your primetime cohort?
Sign up, register an LL-HLS / CMAF probe, and the same edge.edge_partial_segment_availability_ratio · segment_partial_age_ms · cohort.cohort_partial_segment_404_ratio probes that produced the timeline above run on every refresh — and surface in a Slack channel, a webhook, or the streams dashboard.
Synthetic Incident — This scenario uses simulated telemetry constructed from documented streaming behaviors. It does not represent a Streamwake customer outage.
- Pick a recent on-call incident — manifest stall, edge miss, player-side stall, or peer congestion.
- We replay it through the same reliability-agent probe cascade used on the postmortem above.
- You walk away with a written what-could-have-been-Automated readout, not a sales deck.
Read the next postmortem
The other live entries cover HLS/DASH manifest drift on the mid-window packager anchor, DRM license-server cold-starts on a primetime license-fetch stall, and SCTE-35 cue-loss on the live→VOD stitch seam — the four incidents cover the four failure-mode lanes Streamwake reliability agents are tuned for.