When the failover is the failure.
An encoder-cascade postmortem.
A working postmortem on the encoder failover cascade — a primary encoder fails on boot-up or stable ingest, the redundancy contract engages the backup, the backup rejects the upstream ingest, the manifest carries an #EXT-X-DISCONTINUITY tag (or its DASH equivalent) that the players do not silently smooth across, and viewers see player-impacting segment gaps — first-frame freeze on the affected cohort, ABR ladder re-aggregate probe failing across renditions. The Streamwake reliability agent classified it as encoder_failover_cascade · dominant at 87% confidence with primary_encoder_boot_failure_only ruled out by name on the backup ingest rejecting under the contract AND manifest_discontinuity_only ruled out as a tail of the cascade. The remediation policy is authorization-tiered: Tier 0 operator-tier autonomous under gate rests the primary encoder, Tier 1 platform-SRE surfaces the manifest-side stitch, Tier 2 control-plane owner signs off on backup- encoder routing forward — recovery verified viewer-side across T+30 s → T+15 m (first viewer-known-good first frame → sustained viewer-known-good → cohort-known-good → full probe-cohort re-anchor).
Book a technical demo for Encoder failover cascade
Read the postmortem — then bring your own incident to Streamwake.
Two ways to engage on this exact failure pattern: book a 30-minute technical demo where we walk through the probe cascade on your source, or hand us an archived incident and watch the agent diagnose it end-to-end.
Both routes land on the scoping intake form — no SDR gate.
What the cascade shape looks like
Five facts about the cascade shape. Each is pattern-level (the encoder-failover- cascade lane) and not customer-specific — the disclosure above applies to every quantitative claim on this page. Read the bullets in order: trigger, mechanism, seam, viewer impact, recovery cadence.
- 01Primary encoder fails on boot-up or stable ingest — encoder_failover_signal fires the redundancy switch.
- 02Backup encoder rejects the upstream ingest — reject_reason_code reads (network_disable_after_back_to_back_failures | upstream_codec_profile_mismatch | origin_replica_unavailable).
- 03Manifest carries an #EXT-X-DISCONTINUITY (HLS) or a Period discontinuity (DASH) that the players do not silently smooth across.
- 04Cohort sees player-impacting segment gaps — first-frame freeze on the affected viewer cohort, ABR ladder re-aggregate probe failing on 240p / 540p / 720p / 1080p.
- 05Recovery is verified viewer-side: first viewer-known-good first frame (T+30 s) → sustained viewer-known-good for a window (T+2 m) → cohort-known-good across the probe cohort (T+5 m) → full probe-cohort re-anchor (T+15 m).
Trigger: primary encoder fails. Mechanism: backup rejects AND manifest carries a discontinuity tag. Cohort gap: viewers see first-frame freeze and ABR ladder re-aggregate failures. The back_to_back_failuresrejection mode is a documented binding on some failover contracts — when the primary fails, the contract opens the redundancy path, AND a cluster-operator policy says "reject until the cluster stabilises", the backup rejects the upstream ingest, AND the manifest carries the seam between primary's leg and backup's rejected leg.
Why the failover is the failure.
The detector site and the cause are not the same system. On this incident, the cohort segment gap raised the alarm (cohort.cohort_first_frame_freeze_ratio is on the viewer side), the manifest discontinuity + backup rejection was the mechanism (the continuity surface is mid-stack), and the primary encoder failure + failover control plane routing was the cause. Calling it "a primary encoder problem" or "a manifest tag" problem both reach for the wrong remediation lane — only the authorization-tiered policy that addresses all three legs of the cascade closes it.
The cohort reports the segment gap to the player surface. First-frame freeze reads cohort.cohort_first_frame_freeze_ratio = 0.187 vs 0.011 baseline on the affected cohort; the affected cohort cannot produce a known-good first frame (cohort.cohort_first_frame_known_good reads fail) and the ABR ladder re-aggregate probe fails across 240p / 540p / 720p / 1080p. This is the alarm site. The cohort is downstream. The cohort is not the failure.
The manifest carries an #EXT-X-DISCONTINUITYtag between primary's leg and the backup's rejected leg; the cohort reads the tag, the player does not silently smooth across it (the continuity_signal.playback_engine_smoothing_accepted_tag reads false). The backup rejects the upstream ingest (encoder.backup.rejection_reason_code reads network_disable_after_back_to_back_failures). The mechanism is mid-stack — fan-in (cohort & encoder) both read it.
The primary encoder fails on boot-up or stable ingest — and the failover control plane routing is the kind that rejects on back_to_back_failures. The cause is upstream of the mechanism. Authoring the cause lane requires (a) rest of the primary outside the cooldown gate under a cohort signal AND (b) a control-plane owner sign-off forward on the backup-encoder routing — OR an escalation to the encoder vendor if the rejection mode is vendor-bound.
Causal compression across three layers — cohort segment gap (alarm), manifest continuity + backup rejection (mechanism), primary encoder + failover control plane (cause) — is the load-bearing failure shape on this incident. Pinning the closure to the operator-tier lane alone leaves the manifest seam and the cohort gap open. Pinning the closure to the SRE-tier stitch alone strips the tag but does not unblock the backup ingest. Pinning the closure to the cohort side alone decodes the player signal but does not change the manifest tail. The cascade closes only across all three tiers under authorization-aware approval gates.
Symptom remediation vs cascade remediation.
On this incident, two remediation lanes diverge. The symptom lane restarts the player or waits out the cooldown — momentary relief at best. The cascade lane rests the primary outside the cooldown gate under a cohort signal, reanchors the manifest continuity surface under the same signal, AND signs off forward on the backup-encoder routing decision with the control-plane owner — addressing primary + backup + continuity + cohort, together, in cascade. Pick the wrong lane and the cohort gap recurs on the next failover cycle.
Restart the player surface, wait out the primary cooldown, ask the players to smooth the discontinuity tag on the manifest side. This re-creates the cascade on the next failover cycle because the backup ingest is still rejecting under the network_disable_after_back_to_back_failures policy, the manifest still carries the seam, and the next failover cycle lands on the same cohort on the same primary-fail + backup-reject shape.
The player reports momentarily green; the cohort's first-frame freeze clears for one cycle; the manifest seam remains, the backup ingest remains rejecting under the contract, and the next failover cycle lands the cohort gap again — same alarm on the same cohort on the next primary-fail.
Three lanes, three authorizations: Tier 0 operator-tier autonomous under gate rests the primary encoder outside the cooldown gate via the cohort signal; Tier 1 platform-SRE approves the manifest-side stitch under the same cohort signal; Tier 2 control-plane owner signs off on the backup-encoder routing decision forward. Tier 3 operator-team pins the new continuity-signal probe into the cohort fan-in for the next two windows.
The primary encoder boots cleanly outside the cooldown; the backup ingest reads honest under the reconfigured routing; the manifest continuity surface is re-pinned; the next failover cycle lands on a closed cause-tail rather than on the same primary-fail + backup-reject + continuity-tag shape. Verification (T+30 s → T+15 m) closes cohort-side AND encoder-side.
The Streamwake lane on this incident is authorization-tiered cascade remediation + staged viewer-level recovery verification, NOT RCA-only observability. Observability tells you what happened; observability alone does not rest a primary encoder, does not re-publish a manifest, does not change a failover control plane routing. The work that closes the lane is the three-tier remediation (operator + SRE + control- plane owner) under explicit gates, with verification at the viewer level across the staged T+30 s → T+15 m window — not on encoder-side boot green alone, not on the manifest-seam-strip alone.
Each remediation lands under an explicit authorization lane.
The cascade remediation policy is split into four authorization lanes — Tier 0 operator-tier (autonomous under gate), Tier 1 platform-SRE-tier (surfaced for approval), Tier 2 control-plane-owner (sign-off forward / vendor escalation), and Tier 3 operator-team (learn-loop codification). Each tier owns one tail of the cascade. Skipping a tier reopens the cascade on the next failover cycle.
Gating: top_confidence >= 0.80 AND encoder.primary.boot_outcome == fail AND cohort.cohort_first_frame_known_good == fail. Lane: the agent rests the primary encoder and pulls the dispatch outside the cooldown gate via the cohort signal. The operator-tier lane closes first because it is autonomous under the gate.
Gating: continuity_signal.cohort_discontinuity_tag_present == true AND top_confidence >= 0.80. Lane: the platform SRE approves reanchor_manifest_continuity_under_cohort_signal because the manifest continuity surface crosses manifest-publishing ownership scope. Evidence attached: manifest-anchor artifact AND cohort-signal evidence (the cohort is the read side of the seam).
Gating: encoder.backup.rejection_reason_code ∈ documented_rejection_modes AND incident_resolved == true. Lane: the control-plane owner decides — (a) reconfigure the failover routing so network_disable_after_back_to_back_failures is not the binding dispatch on the next engagement, OR (b) escalate to the encoder vendor if the rejection mode is vendor-bound. The agent does not emit the routing patch on its own — the cause tail of the cascade is sign-off forward.
Gating: incident_resolved == true AND cohort-side close-out signals cleared AND continuity_signal.cohort_discontinuity_tag_present_probe_fan_in_pinned_for_next_two_windows == false. Lane: operator-team pins the new continuity-signal probe into the cohort probe fan-in for the next two windows so the false-positive rate can be measured before the probe is promoted.
- Tier 0 acts on its own under a confidence + cohort-signal gate. The lane is bounded by
rest_primary_encoder_and_pull_dispatch_outside_cooldown_gate_via_cohort_signal— the operator-tier acts only inside this gate. - Tier 1 surfaces — the platform SRE owns the manifest-publishing continuity scope; the agent surfaces the reanchor under the cohort signal with evidence on both sides.
- Tier 2 surfaces — the control-plane owner owns the failover control-plane routing decision; on a vendor-bound rejection, escalation to the encoder vendor is the explicit forward lane.
- Tier 3 surfaces — the operator team owns the cohort probe fan-in cadence; pinning the new probe is measured before promotion.
Without infrastructure-green, recovery reads at the viewer level — T+30 s → T+15 m.
Recovery on this incident is staged viewer-level — T+30 s first viewer-known-good first frame → T+2 m sustained viewer-known-good window → T+5 m cohort-known-good across the probe cohort → T+15 m full probe-cohort re-anchor ahead of the next primetime. Not on infrastructure-green (encoder-side boot_pass alone is not recovery), not on continuity-strip alone (manifest-side stitch alone is not recovery), not on backup-ingest-acceptance alone (the backup leg alone does not prove the cohort gap has settled).
cohort.cohort_first_frame_known_good reads pass on the primary cohort within T+30 s of the Tier 0 rest landing with evidence attached. The first viewer-known-good first frame clears the operator- tier leg of the closure — the cohort reads known-good on the fresh primary cohort.
cohort.cohort_first_frame_freeze_ratio settles to within baseline; continuity_signal.cohort_discontinuity_tag_present reads false on the affected cohort. The mechanism tail of the cascade is closed: the manifest carries no seam on the cohort side, and the player reads green.
cohort.cohort_ladder_rendition_reaggregate_probe reads pass on every monitored rendition (240p / 540p / 720p / 1080p); cohort.cohort_join_stall_ratio settles ≤ 0.05 within ten consecutive 30 s windows on the affected cohort.
cohort.cohort_rebuffer_ratio settles within baseline; cohort.cohort_startup_time_p95_ms reads ahead of the next primetime window. The cascade is closed across the cohorts that read it.
encoder.primary.boot_outcome returning to pass is NOT a close-out signal — passes the operator-tier leg, does not prove the backup ingest or the continuity surface is closed. continuity_signal.cohort_discontinuity_tag_present reading false is NOT a close-out signal — passes the SRE-tier mechanism leg, does not prove the cohort gap has settled. encoder.backup.ingest_acceptance.post_failover reading 1+ accepted is NOT a close-out signal — passes the backup-leg, does not prove the cohort reads the continuity as known-good. Recovery is staged viewer-side.
Anatomy of the evidence packet
Two packets on this incident — a synthetic encoder/manifest/cohort probe packet that surfaces the three-layer cascade (primary encoder fail → failover contract engaging the backup → backup rejection → cohort segment gap) with the authorization-tiered gate evidence attached, and a synthetic Streamwake-direction trace of the response loop (ranked hypotheses, causal compression across three layers, authorization-tiered governed actions, staged viewer-level recovery verification). Both packets are tagged direction: 'synthetic'.
GET /live/event/stream.m3u8 HTTP/1.1
host: encoder-cohort.ops.example.com
accept: application/vnd.apple.mpegurl
direction: synthetic # simulated telemetry against a documented failover-vector shape
----- cycle 0 (T+0m, primary encoder failing — failover trigger) -----
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=58
x-encoder-primary-cohort: encoder-prime-04 ← the primary that started failing
x-encoder-region: us-east-2
x-packager: pkg-prime-04
direction: synthetic # this packet is reconstructed against documented shapes
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:84217
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000,
084217.ts
# encoder.primary.boot_outcome: fail (primary encoder boot_up or stable_ingest fails)
# encoder.primary.boot_error_code: {PEER-DETACH, CODEC-INIT, INTERNAL-SW-CODEC-MISS,
# CODEC-PROFILE-MISMATCH, RTC-NETWORK-DISABLE,
# HW-TRANSCODER-DEGRADED} (primary-side error; orthogonal to the redundancy contract)
# encoder.primary.failover_signal: fire (the failover contract opened the redundancy path)
# encoder.primary.boot_cycles_since_8s_window: 312 (primary cycling; this is the alarm site, not the failure site)
# encoder.primary.cooldown_state: armed-but-not-cleared (the cooldown gate has not lifted cleanly)
----- cycle 1 (T+~30 s, backup encoder engaged — backup rejection) -----
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=64
x-encoder-backup-cohort: encoder-backup-04 ← the backup the contract engaged
x-encoder-failover-mode: backup-redundancy-v2
x-encoder-failover-issued-at-t: 0s
direction: synthetic # this packet is reconstructed against documented shapes
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:84217
#EXT-X-MAP:URI="init.mp4"
#EXT-X-DISCONTINUITY ← the manifest carries a discontinuity tag
#EXTINF:6.000,
084217.ts
# encoder.backup.boot_outcome: fail (backup rejects the upstream ingest)
# encoder.backup.boot_error_code: BACKUP-UPSTREAM-REJECT
# encoder.backup.rejection_reason_code: network_disable_after_back_to_back_failures
# (the cluster-operator policy the backup is bound by;
# a documented rejection mode in some vendor failover contracts)
# encoder.backup.rejection_mode_observed_times: 3 (the backup rejected three back-to-back attempts within 4 s)
# encoder.backup.ingest_acceptance.post_failover: 0 accepted (no segment legs landed on the backup)
# encoder.failover_signal_history: [primary.fail, failover.fire, backup.reject,
# continuity_tag.present, cohort.first_frame.fail]
# encoder.failover_cooldown_state: armed-with-rejection-override (cooldown gate held; the
# override did not lift cleanly because the backup rejected)
----- cycle 2 (T+~60 s, on the affected viewer cohort — segment gap surfaces) -----
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=132
x-cdn: cdn-A/atl01
x-pop: cdn-A/atl01
x-multicdn-route: cdn-A
direction: synthetic # this packet is reconstructed against documented shapes
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:84217
#EXT-X-MAP:URI="init.mp4"
#EXT-X-DISCONTINUITY ← the discontinuity tag that the cohort is now reading
#EXTINF:6.000,
084217.ts
# cohort.cohort_first_frame_known_good: fail (the affected cohort cannot produce a known-good first frame)
# cohort.cohort_first_frame_freeze_ratio: 0.187 (vs 0.011 baseline, 90 s window — first-frame freeze on the affected cohort)
# cohort.cohort_ladder_rendition_reaggregate_probe: fail (240p / 540p / 720p / 1080p re-aggregate probe
# fails across renditions on the affected cohort — the
# discontinuity tag breaks rendition-level stitch)
# cohort.cohort_join_stall_ratio: 0.142 (vs 0.018 baseline — late-join stalled behind the gap)
# cohort.cohort_rebuffer_ratio: 0.084 (vs 0.012 baseline — playback stalled on the affected cohort)
# cohort.cohort_startup_time_p95_ms: 7810 (vs 1840 baseline — TTFF lifted above the 4 s threshold)
# cohort.cohort_affected_cohort_id: apac-teal-cascade-affected-cohort
# continuity_signal.cohort_discontinuity_tag_present: true (the cohort is reading the discontinuity tag the manifest carries)
# continuity_signal.affected_renditions: [240p, 540p, 720p, 1080p]
# continuity_signal.ext_x_discontinuity_seen_on_renditions_count: 4
# continuity_signal.segment_leg_cache_hit: pass (segment leg green — the failure is on continuity, not on cache-miss posture)
# continuity_signal.playback_engine_smoothing_accepted_tag: false (the player does not silently smooth across
# the discontinuity tag — the gap reaches the viewer)
----- cycle 3 (T+~12 m, after Tier 0 dispatch + Tier 1 manifest-side stitch + Tier 2 backup-routing ack) -----
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=128
x-cdn: cdn-A/atl01
x-multicdn-route: cdn-A
direction: synthetic # this packet is reconstructed against documented shapes
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:84223
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000,
084223.ts
# encoder.primary.boot_outcome: pass (primary booted cleanly under the Tier 0 dispatch)
# encoder.primary.cooldown_state: cleared (cooldown gate lifted cleanly on a fresh probe cycle)
# encoder.backup.ingest_acceptance.post_failover: 1 accepted (segment leg landed; backup read honest under the new routing)
# continuity_signal.cohort_discontinuity_tag_present: false (manifest continuity re-pinned; the tag is gone)
# cohort.cohort_first_frame_known_good: pass (viewer-known-good first frame on the primary cohort — T+30 s gate cleared)
# cohort.cohort_first_frame_freeze_ratio: 0.014 (within cohort baseline — sustained viewer-known-good window cleared)
# cohort.cohort_ladder_rendition_reaggregate_probe: pass (240p / 540p / 720p / 1080p re-aggregate pass — cohort-known-good cleared)
# cohort.cohort_join_stall_ratio: 0.022 (within cohort baseline)
# cohort.cohort_rebuffer_ratio: 0.013 (within cohort baseline)
# cohort.cohort_startup_time_p95_ms: 1940 (within startup-time baseline — ahead of the next primetime window)
# governed_action_emitted: rest_primary_encoder_and_pull_dispatch_outside_cooldown_gate_via_cohort_signal,
# reanchor_manifest_continuity_under_cohort_signal,
# surface_backup_encoder_routing_decision_to_control_plane_owner_for_sign_off_forward,
# surface_continuity_signal.cohort_discontinuity_tag_present_probe_to_cohort_fanin- encoder.primary.boot_outcome →
fail (the trigger; the failover signal fires) - encoder.primary.failover_signal →
fire (redundancy contract engaged the backup) - encoder.backup.rejection_reason_code →
network_disable_after_back_to_back_failures (the documented binding rejection mode) - encoder.backup.ingest_acceptance.post_failover →
0 accepted (no segment legs landed on the backup) - continuity_signal.cohort_discontinuity_tag_present →
true (the manifest carries an #EXT-X-DISCONTINUITY tag the cohort reads) - cohort.cohort_first_frame_known_good →
fail (the cohort cannot produce a known-good first frame) - cohort.cohort_ladder_rendition_reaggregate_probe →
fail across 240p / 540p / 720p / 1080p (the seam breaks rendition-level stitch) - continuity_signal.playback_engine_smoothing_accepted_tag →
false (the player does NOT silently smooth across the seam — the gap reaches the viewer)
{
"stream_id": "ckencoderfailovercascade7142",
"source": "https://encoder-cohort.ops.example.com/live/event/stream.m3u8",
"direction": "synthetic", # simulated telemetry against a documented failover-vector shape
"checked_at": "2026-08-23T18:42:11Z",
"context": "Synthetic postmortem — encoder failover cascade (primary fail → backup reject → manifest discontinuity → cohort segment gap). Streamwake was not involved in any incident matching this fingerprint; this trace is reconstructed from documented encoding/failover behaviors to illustrate how Streamwake would process the same shape.",
"ranked_hypotheses": [
{
"rank": 1,
"tag": "dominant",
"hypothesis": "encoder_failover_cascade",
"confidence": 0.87,
"evidence_signals": [
"encoder.primary.boot_outcome → fail (primary boot_up or stable_ingest fails; the failover contract opens the redundancy path)",
"encoder.primary.failover_signal → fire (the contract is engaged; this is the load-bearing primary-side signal)",
"encoder.backup.rejection_reason_code → network_disable_after_back_to_back_failures (a documented rejection mode the backup is bound by — the backup rejected three back-to-back attempts within 4 s)",
"encoder.backup.ingest_acceptance.post_failover → 0 accepted (no segment leg landed on the backup under the redundancy contract)",
"continuity_signal.cohort_discontinuity_tag_present → true (the manifest carries an #EXT-X-DISCONTINUITY tag the cohort reads)",
"continuity_signal.playback_engine_smoothing_accepted_tag → false (the player does not silently smooth across the discontinuity tag — the gap reaches the viewer)",
"cohort.cohort_first_frame_known_good → fail (viewer-known-good first frame fails on the affected viewer cohort)",
"cohort.cohort_ladder_rendition_reaggregate_probe → fail across 240p / 540p / 720p / 1080p (the discontinuity tag breaks rendition-level stitch on the affected cohort)",
"cohort.cohort_first_frame_freeze_ratio → 0.187 vs 0.011 baseline (first-frame freeze on the affected viewer cohort)"
],
"explicit_note": "primary-failure-only does NOT explain the cascade. The cascade is shaped by primary fail + backup reject + manifest discontinuity + cohort gap, together — a primary fix that does not also unblock the backup ingest or repin manifest continuity leaves the cohort gap untouched."
},
{
"rank": 2,
"tag": "ruled_out_by",
"hypothesis": "primary_encoder_boot_failure_only",
"confidence": 0.22,
"evidence_signals": [
"encoder.primary.boot_outcome → fail — but the cohort gap remains after rest_primary_encoder if the backup ingest is still rejecting under the failover contract",
"if primary were the only failure site, encoder.backup.ingest_acceptance.post_failover would read 1+ accepted — it reads 0 accepted; the cascade is not closed by a primary-rest alone",
"the manifest discontinuity tag remains present after primary-rest if the manifest continuity surface has not been re-anchored under a cohort signal — the cohort gap propagates"
],
"dismissed_reason": "calling it 'a primary encoder problem' lets the cascade through to the viewer cohort. The primary side is the failure site for ONE leg of the cascade; it is not the close-out site."
},
{
"rank": 3,
"tag": "ruled_out_by",
"hypothesis": "manifest_discontinuity_only",
"confidence": 0.18,
"evidence_signals": [
"manifest continuity alone does not produce the cohort gap if the backup is actually accepting ingest — the cohort gap is on player reading the tag, but the tag is a consequence, not a cause",
"if the continuity tag alone were the cause, rest_primary_encoder would not lift cohort.cohort_first_frame_known_good — but it does when paired with reanchor_manifest_continuity_under_cohort_signal",
"dismissed — manifest discontinuity is one of three tails of the cascade; treating it alone reverses the lane"
],
"dismissed_reason": "manifest continuity is a tail of the cascade, not the cause. Re-anchoring continuity on top of an unfixed primary + unfixed backup ingest just hides the gap behind a tag-strip that doesn't reach the cohort."
}
],
"causal_compression": [
"cohort segment gap on the player (alarm site)",
"manifest discontinuity + backup rejection (mechanism)",
"primary encoder fail + failover control-plane routing (cause)"
],
"governed_actions": [
{
"action": "rest_primary_encoder_and_pull_dispatch_outside_cooldown_gate_via_cohort_signal",
"type": "governed",
"decision_lane": "autonomous_under_gate",
"authorization_tier": "Tier 0 (operator-tier · autonomous under gate)",
"gating": "top_confidence >= 0.80 AND encoder.primary.boot_outcome == fail AND cohort.cohort_first_frame_known_good == fail",
"evidence": "top-hypothesis confidence 0.87; encoder.primary.boot_outcome = fail; cohort_first_frame_known_good = fail",
"expected_effect": "encoder.primary.boot_outcome → pass under a fresh probe cycle; encoder.primary.cooldown_state → cleared outside the cooldown gate via the cohort signal — the operator-tier lane is closed"
},
{
"action": "reanchor_manifest_continuity_under_cohort_signal",
"type": "governed",
"decision_lane": "surfaced_to_humans",
"authorization_tier": "Tier 1 (platform-SRE-tier · surfaced to SRE)",
"gating": "continuity_signal.cohort_discontinuity_tag_present == true AND top_confidence >= 0.80",
"evidence": "cohort_discontinuity_tag_present = true; top-hypothesis confidence 0.87; surfaced to platform-SRE because the manifest continuity surface crosses manifest-publishing ownership scope",
"expected_effect": "continuity_signal.cohort_discontinuity_tag_present → false on the affected cohort within the staged T+2 m sustained viewer-known-good window — the mechanism tail of the cascade is closed under RTT-tagged reanchor"
},
{
"action": "surface_backup_encoder_routing_decision_to_control_plane_owner_for_sign_off_forward",
"type": "governed",
"decision_lane": "surfaced_to_humans_for_approval_forward",
"authorization_tier": "Tier 2 (control-plane-owner · sign-off forward)",
"gating": "encoder.backup.rejection_reason_code ∈ documented_rejection_modes AND incident_resolved == true",
"evidence": "encoder.backup.rejection_reason_code = network_disable_after_back_to_back_failures (a documented vendor/cluster-operator policy); the control-plane owner signs off on whether the failover contract routing is for this rejection mode OR whether to escalate to the encoder vendor",
"expected_effect": "control-plane owner decides: (a) reconfigure the failover routing so that rejection-mode 'network_disable_after_back_to_back_failures' is not the binding dispatches on the next engagement, OR (b) escalate to the encoder vendor if the rejection is from a vendor policy the customer does not control — the cause tail of the cascade is closed"
},
{
"action": "surface_continuity_signal.cohort_discontinuity_tag_present_probe_to_cohort_fanin",
"type": "governed",
"decision_lane": "surfaced_to_humans_for_approval_forward",
"authorization_tier": "Tier 3 (operator-team · sign-off forward · learn-loop codification)",
"gating": "incident_resolved == true AND cohort-side close-out signals cleared AND continuity_signal.cohort_discontinuity_tag_present_probe_fan_in_pinned_for_next_two_windows == false",
"evidence": "cohort-side close-out signals (cohort_first_frame_known_good + cohort_sustained_known_good_window + cohort_known_good_across_probe_cohort + cohort.ladder_rendition_reaggregate_probe) cleared within tolerance; the operator team pins the new probe into the cohort fan-in for the next two windows before promotion",
"expected_effect": "the continuity-signal probe is codified into the cohort fan-in for the next two windows so the false-positive rate can be measured before the probe is promoted"
}
],
"verification_window": {
"close_out_signal": "viewer_level_staged_recovery",
"stages": [
"T+30s — first viewer-known-good first frame on the primary cohort (cohort.cohort_first_frame_known_good → pass)",
"T+2m — sustained viewer-known-good across a window (cohort.cohort_first_frame_freeze_ratio within baseline; continuity_signal.cohort_discontinuity_tag_present → false)",
"T+5m — cohort-known-good across the probe cohort (cohort.cohort_ladder_rendition_reaggregate_probe → pass; cohort.cohort_join_stall_ratio within baseline)",
"T+15m — full probe-cohort re-anchor (cohort.cohort_rebuffer_ratio within baseline; cohort_startup_time_p95_ms ahead of the next primetime window)"
],
"NOT_close_out_signal": [
"encoder.primary.boot_outcome → pass alone — passes the operator-tier leg, does not prove the cascade mechanism is closed",
"continuity_signal.cohort_discontinuity_tag_present → false alone — passes the SRE-tier mechanism leg, does not prove the cohort gap is settled",
"encoder.backup.ingest_acceptance.post_failover = 1 accepted alone — passes the backup-leg, does not prove the cohort reads the continuity as known-good"
],
"explicit_note": "fan-in-green is NOT recovery. Recovery is staged viewer-level (T+30s → T+15m)."
},
"surfaced_to_humans": [
{"owner": "platform SRE", "task": "approve the manifest-side reanchor under the cohort signal — the continuity surface crosses manifest-publishing ownership scope (Tier 1)"},
{"owner": "control-plane owner", "task": "decide the backup-encoder routing sign-off forward: reconfigure OR vendor-escalate, depending on whether the rejection mode is a customer-controlled cluster-operator policy OR a vendor binding policy (Tier 2)"},
{"owner": "encoder vendor (escalation)", "task": "if the rejection mode is vendor-bound, the encoder vendor provides the routing policy / failover contract revision for the next engagement (Tier 2 vendor-owner)"},
{"owner": "operator-team", "task": "sign off on pinning continuity_signal.cohort_discontinuity_tag_present into the cohort probe fan-in for the next two windows (Tier 3 — learn-loop codification)"},
{"owner": "on-call", "task": "page on the encoder_failover_cascade root-cause review — primary fail + backup reject + manifest discontinuity + cohort gap, addressed in authorization-tiered policy"},
{"owner": "reliability team", "task": "assign the public postmortem write-up (this page)"}
]
}- rank 1 →
encoder_failover_cascade · 0.87 (primary fail + backup reject + manifest discontinuity + cohort gap, in cascade) - rank 2 →
primary_encoder_boot_failure_only · 0.22 (dismissed — primary fix alone leaves backup ingesting nothing on the next failover) - rank 3 →
manifest_discontinuity_only · 0.18 (dismissed — continuity is a tail of the cascade, not the cause) - tier 0 →
rest_primary_encoder_and_pull_dispatch_outside_cooldown_gate_via_cohort_signal(operator-tier · autonomous under gate) - tier 1 →
reanchor_manifest_continuity_under_cohort_signal(platform-SRE-tier · surfaced for SRE approval) - tier 2 →
surface_backup_encoder_routing_decision_to_control_plane_owner_for_sign_off_forward(control-plane owner · reconfigure OR escalate to encoder vendor) - close-out signal → staged viewer-level:
cohort.cohort_first_frame_known_good(T+30s) + sustained viewer-known-good (T+2m) +cohort.cohort_ladder_rendition_reaggregate_probe(T+5m) + full probe-cohort re-anchor (T+15m) — NOT infrastructure-green alone
The seven-step response loop on this incident.
Seven steps close the lane on an encoder-failover cascade of this shape. The loop is structural, not compressed — Detect (alarm on the cohort side AND the encoder side), Investigate (fan across three layers), Decide (rank the top hypothesis), Approve (authorization-tiered: Tier 0 / Tier 1 / Tier 2 / Tier 3), Act (gated, not autonomous outside the gate), Verify (staged viewer-level T+30 s → T+15 m), Learn (codify the cascade pattern). The probe packets and Streamwake-direction trace above are synthetics — not a real incident.
The detector fans across the cohort side AND the encoder side. The primary encoder cycles on boot (encoder.primary.boot_outcome reads fail across the affected cohort boundary), the failover contract opens the redundancy path (encoder.primary.failover_signal fires), the backup rejects the upstream ingest (encoder.backup.rejection_reason_code reads network_disable_after_back_to_back_failures), the manifest carries an #EXT-X-DISCONTINUITY tag the cohort reads (continuity_signal.cohort_discontinuity_tag_present reads true), AND the cohort reports a first-frame freeze (cohort.cohort_first_frame_freeze_ratio lifted to 0.187 vs 0.011 baseline). The cohort side AND the encoder side, together, are the alarm site.
The investigation fans across three layers, in order: encoder side (primary fail + backup reject), control-plane / failover routing (the contract that engaged the backup), and manifest continuity surface (#EXT-X-DISCONTINUITY tag present on the manifest, the cohort reads it). The discriminator against a primary-only incident is that rest_primary_encoder alone leaves the backup ingest rejecting if the failover control plane is unchanged — the cascade is not closed by one leg. The discriminator against a continuity-only incident is that stripping the tag without unblocking the backup ingest just hides the gap behind a tag-strip that does not reach the cohort.
Three ranked hypotheses. The top hypothesis — encoder_failover_cascade — files at 0.87 confidence on primary fail + backup reject + manifest discontinuity + cohort gap, together. primary_encoder_boot_failure_only is held at 0.22 because a primary fix that does not unblock the backup ingest does not close the cascade. manifest_discontinuity_only is held at 0.18 because manifest continuity is a tail of the cascade, not the cause. The shape matters — primary + backup + continuity + cohort, in cascade, is the lane.
Three lanes, three approvals. Tier 0 surfaces the rest_primary_encoder_and_pull_dispatch_outside_cooldown_gate_via_cohort_signal under an explicit gate — top_confidence >= 0.80 AND encoder.primary.boot_outcome == fail AND cohort.cohort_first_frame_known_good == fail; agent acts on its own if the gate clears. Tier 1 surfaces reanchor_manifest_continuity_under_cohort_signal to platform-SRE — the manifest continuity surface crosses manifest-publishing ownership scope and is owner-aware. Tier 2 surfaces surface_backup_encoder_routing_decision_to_control_plane_owner_for_sign_off_forward to the control-plane owner — the backup-encoder routing decision is sign-off forward because rejection_mode ∈ documented_rejection_modes AND the decision (reconfigure OR vendor-escalate) is not the agent's to make.
The act is split. Tier 0 — rest_primary_encoder_and_pull_dispatch_outside_cooldown_gate_via_cohort_signal — lands inside the cohort signal: the operator-tier lane is autonomous under the gate (the gate cleared on a fresh probe cycle). Tier 1 — reanchor_manifest_continuity_under_cohort_signal — lands after SRE approves with continuity evidence attached (the manifest-anchor artifact AND the cohort-signal evidence). Tier 2 — surface_backup_encoder_routing_decision_to_control_plane_owner_for_sign_off_forward — is forward-routed: the control-plane owner decides, the agent does not emit a backup-routing patch on its own. No probe-driven rebalance masks the cascade.
The verification window closes in four stages, viewer-side, NOT infrastructure-side. T+30 s — cohort.cohort_first_frame_known_good reads pass on the primary cohort (first viewer-known-good first frame). T+2 m — cohort.cohort_first_frame_freeze_ratio within baseline AND continuity_signal.cohort_discontinuity_tag_present reads false (sustained viewer-known-good across a window). T+5 m — cohort.cohort_ladder_rendition_reaggregate_probe reads pass on every monitored rendition (cohort-known-good across the probe cohort). T+15 m — cohort.cohort_rebuffer_ratio within baseline AND cohort_startup_time_p95_ms reads ahead of the next primetime window (full probe-cohort re-anchor). Infrastructure-green is NOT recovery — recovery is staged viewer-level.
The learn step codifies the pattern forward: any production shape where (primary encoder fail) + (backup encoder reject) + (manifest discontinuity tag present) + (cohort segment gap) is observed together is the encoder_failover_cascade failure shape — primary-only incidents do not produce a cohort gap if the backup is accepting ingest on the next failover cycle, and continuity-only incidents do not produce a cohort gap if the tag is stripped without breaking the cohort. The cohort side AND the encoder side, together, are the alarm site. Tier 0 rests the operator-tier lane; Tier 1 lands the SRE-tier mechanism tail; Tier 2 closes the control-plane cause tail.
Authorization-tiered cascade remediation.
Staged viewer-level recovery verification.
Streamwake's lane on this incident is authorization-tiered remediation across four tiers (operator + platform-SRE + control-plane owner + operator-team-learnings) AND staged viewer-level recovery verification (T+30 s → T+15 m) — not RCA-only observability, not single-signal recovery, not probe-driven rebalances that mask the cascade. The position slot here is fixed: respect the lane of each authorization tier AND verify at the viewer level. The Streamwake reliability agent does not claim to have prevented the underlying cascade; the agent is positioned to process the same shape with the cascade-specific authorization gating AND the cohort-side staging.
- Does: detect on the cohort side AND the encoder side across three layers (primary, control-plane / failover routing, manifest continuity); rank the cascade hypothesis (
encoder_failover_cascade); rest the primary under a confidence-and-cohort-signal gate (Tier 0); surface the manifest-side stitch to platform SRE under the same cohort signal (Tier 1); surface the backup-routing decision to the control-plane owner with vendor- escalation as the forward alternative (Tier 2); pin the new continuity-signal probe into the cohort probe fan-in for the next two windows (Tier 3); verify at the viewer level across the staged T+30 s → T+15 m window. - Does not: autonomously emit a manifest-side stitch (the continuity surface crosses manifest-publishing ownership scope and requires Tier 1 approval); autonomously emit a backup-encoder routing patch (the cause tail of the cascade is sign-off forward — Tier 2 control-plane owner decides reconfigure OR escalation); declare the incident resolved on infrastructure- green (recovery is staged viewer-level T+30 s → T+15 m, not on encoder-side boot_pass alone, not on continuity-strip alone); overwrite the cohort-side signal with a feasibility-rerank that masks the cascade.
What the agent codifies forward from this failure shape.
A pattern-level note (not customer-specific) on what the Streamwake reliability agent codifies into its detection / ranking / approval gates from this failure shape. The pattern is: any production shape where primary encoder fail + backup reject + manifest discontinuity + cohort gap is observed together is the encoder_failover_cascade failure lane; primary-only and continuity-only incidents do not produce a cohort gap on the next failover.
The agent codifies encoder_failover_cascade as a discrete failure-mode entry — not a degenerate case of 'primary encoder failure', not a degenerate case of 'manifest discontinuity tag'. The cascade ranks against primary_encoder_boot_failure_only (0.22) and manifest_discontinuity_only (0.18) on every cohort-side signal observed — a primary alarm that does not also present a backup rejection OR a continuity tag is NOT this shape; an infrastructure-green alarm that does not also present a cohort gap is NOT this shape.
The agent codifies cohort.cohort_first_frame_known_good and cohort.cohort_first_frame_freeze_ratio as first-class evidence — NOT summary statistics. The cohort segment gap is the read side of the cascade, and the alarm site is on the player surface. Stripping the cohort signals from the cascade and ending up only with the encoder-side signals (primary fail + backup reject) loses the seam: the seam reaches the viewer via the cohort side.
The agent codifies the four-tier authorization model as observed-on-this-shape: Tier 0 (operator-tier · autonomous under gate) → Tier 1 (platform-SRE · surfaced for approval) → Tier 2 (control-plane owner · sign-off forward · vendor escalation as the forward alternative) → Tier 3 (operator-team · learn-loop codification). Each tier owns one tail of the cascade; skipping a tier re-opens a tail on the next failover cycle.
Primary encoder.primary.boot_outcome = fail + backup encoder.backup.rejection_reason_code ∈ documented_rejection_modes + manifest continuity_signal.cohort_discontinuity_tag_present = true (player does NOT silently smooth) + cohort cohort.cohort_first_frame_known_good = fail. Together, in cascade. Recovery is staged viewer-level: T+30 s → T+15 m. Ruled out: primary-only (0.22), continuity-only (0.18).
Want Streamwake to process an encoder-failover cascade on your fleet?
Sign up, register an encoder cohort probe, and the same encoder.primary.boot_outcome · encoder.primary.failover_signal · encoder.backup.rejection_reason_code · continuity_signal.cohort_discontinuity_tag_present · cohort.cohort_first_frame_known_good · cohort.cohort_ladder_rendition_reaggregate_probe probes that produced the trace above run on every prime-cohort refresh — and surface in a Slack channel, a webhook, or the streams dashboard.
Synthetic Incident — This scenario uses simulated telemetry constructed from documented encoding/failover behaviors. It does not represent a Streamwake customer outage.
- Pick a recent on-call incident — manifest stall, edge miss, player-side stall, or peer congestion.
- We replay it through the same reliability-agent probe cascade used on the postmortem above.
- You walk away with a written what-could-have-been-Automated readout, not a sales deck.
Related writeups
The closest siblings cover player-visible symptom lands and authorization-tiered remediation policy that frame streamwake reliability agents — origin shield queue saturation under a marquee live-event spike and the correlated cache-miss storm variant, ABR package-list drift on a mid-stream CDN failover, ISP-vs-CDN triaging, and the encoder-isn't-the-failure cross-layer response. Together they cover the cohort-side seams a cascade-incision investigation fans across when the failover is the failure.