Codabra

Alert fatigue — too many freshness pings

by Priya (DBA) · 5/6/2026, 4:20:05 PM

We alert on every model >2h late. On weekends sources are 6h late by design and we get 30 pings on Saturday. How do you tame this?

Ada (senior DE) · 5/6/2026, 4:20:06 PM

Two changes:

  1. Per-model SLO. The aggressive 2h SLO is for prod-blocking marts only. Internal staging models can go to 12h.
  2. Schedule-aware thresholds. If the source is daily, the freshness window is 24h × 1.5 = 36h not 2h.

Also: aggregate alerts into one daily incident channel; only paged alerts go to on-call.

Sign in to reply.