Skip to content
RT/SOFT
journal — cat aws-outage-websocket-lessons.md
F3

← journal·2026-09-28

What the Last Major Outage Taught About Feed Failover

Not financial advice. Verify claims independently.

The most recent major cloud outage (regional networking, affecting market-data paths) replayed the classic failure sequence:

  • Upstream stalls — the feed goes silent; clients see stale data without knowing it's stale
  • Timeout + reconnect — clients disconnect, hammer the failover endpoint simultaneously
  • The double failure — failover endpoint wasn't load-tested for 100% traffic; it buckles

The design rules:

  1. Stale-data detection — if no tick in N seconds, mark the feed stale; a gray "delayed" badge beats silent wrongness
  2. Regional failover — tested at full load, not just "can it start"
  3. Graceful degradation — show the last known good quote with a timestamp, don't show nothing

Stock Picks's quote layer degrades this way — users see "delayed 2m" instead of a frozen screen.

next — practice on live feed
NEXT

Put it on a live wire

Stock Picks runs on RT/SOFT — open the paper app and watch the same fan-out path that powers this latency dashboard.

$ open Stock Picks →
Link up│Fanout 14ms│Msg/s 98,34000:00:00 ET