Postmortems Index

Last updated 2 min read
Report issue

Postmortems#

We publish postmortems for any production incident that:

  • Affected customer-visible behavior for >15 minutes, OR
  • Resulted in lost data, OR
  • Surfaced a bug class we hadn't seen before (even if no customer impact)

The full archive lives in docs/postmortems/ in the repo. Anchored summaries below.

2026-05-14 — Agent crash loop (138 restarts in 24h)#

Severity: SEV-2 (panel-visible flapping, no data loss). Resolution time: ~4 hours (incident found by chance during a routine check). Root cause: Three layered issues — five sync I/O calls in async event-loop functions, co-tenant CPU starvation on the host, and a PYTHONPATH-shadow footgun causing the deployed fix to silently revert. Fix: PR #55 — wrap all sync I/O in asyncio.to_thread, add outer wait_for caps on firewall probes, refactor heartbeat watermark, systemd CPUWeight=300/IOWeight=300/Nice=-5/HEARTBEAT_INTERVAL=120. What we learned: don't trust journalctl-silence as a stall signal; always py-spy dump after deploys; document the PYTHONPATH shadow.

Full postmortem →

Our postmortem template#

Every postmortem follows the same structure:

  • Status / Severity / Impact — at the top so the on-call doesn't have to scroll
  • Timeline (compressed) — minute-by-minute, with timestamps in UTC
  • Root cause — usually multiple, layered; we name each one
  • Why this happened — the conditions that allowed the bug class to exist (process / design / staffing)
  • What we did to fix it — concrete PRs + out-of-band actions
  • What we changed so it can't happen again — the durable prevention
  • What we got lucky about — the things that could have been worse
  • Action items — checkboxed, owned, with dates

This template is in docs/postmortems/README.md.

Why we publish them publicly#

  • Customers using GritivaCore in production should know how we operate under pressure
  • The postmortem template is reusable — feel free to fork it for your own team
  • The patterns we hit (sync I/O on event loop, CPU starvation by co-tenants, the PYTHONPATH shadow) are common — others can avoid them
  • Public accountability is a forcing function for actually completing the action items

See also#