Runbook: Agent Crash Loop

Last updated 2 min read
Report issue

Runbook: Agent Crash Loop#

When you see: agent's NRestarts climbing in systemctl show gritiva-agent, or VM flapping online/offline in the panel every few minutes.

Severity guide: if a single agent is restarting >3/hour, treat as SEV-3. If many agents simultaneously, SEV-2.

Symptoms#

  • NRestarts >0 and increasing in systemctl show gritiva-agent | grep NRestarts
  • Panel shows the VM oscillating between online and offline
  • journalctl -u gritiva-agent shows repeated Stopping… / Started… / Started server process lines
  • Mesh peer state for any scope on this VM may be stuck in CONNECTING

Diagnose#

Step 1 — Confirm the loop#

BASH
sudo systemctl show gritiva-agent | grep -E 'NRestarts|Status|Result'
sudo journalctl -u gritiva-agent --since "30 minutes ago" | grep -c 'Started server process'

If you see >3 in 30 minutes, you're in a crash loop.

Step 2 — Capture a stack with py-spy#

BASH
sudo apt install -y python3-pip && sudo pip3 install py-spy
PID=$(pgrep -f 'gritiva-agent' | head -1)
sudo py-spy dump --pid $PID

Look at MainThread. Healthy = selectors.select (idle in event loop). Bad = psutil.* or subprocess.* or socket.* blocking the event loop.

Step 3 — Check CPU pressure on the host#

BASH
cat /sys/fs/cgroup/system.slice/gritiva-agent.service/cpu.pressure
# look for "some avg10=" — anything >5% means the agent is starved

If some avg10 is high, a co-tenant workload is eating CPU. Common offenders: docker containers with no resource limits, OCR/AI workloads, postgres VACUUM.

Step 4 — Check the heartbeat watchdog log#

BASH
sudo journalctl -u gritiva-agent --since "30 minutes ago" | grep -E 'Heartbeat sent|watchdog'

If you see watchdog exits, the agent's own heartbeat watchdog killed it because a heartbeat send took longer than the deadline (default 510s with HEARTBEAT_INTERVAL=120).

Mitigate (immediate)#

If the host is genuinely under-resourced:

BASH
sudo systemctl set-property gritiva-agent.service   CPUWeight=300 IOWeight=300 Nice=-5   --runtime
sudo systemctl restart gritiva-agent

This bumps the agent's scheduling priority. The change persists until reboot — to make it permanent, edit the systemd unit (this is the post-PR-#55 default for new installs).

Fix (durable)#

The full root-cause set is in the 2026-05-14 postmortem. The TL;DR fixes shipped in PR #55:

  1. All sync I/O calls in async functions now wrapped in asyncio.to_thread
  2. Outer wait_for(20s) cap on every firewall probe
  3. Heartbeat watermark only stamped on confirmed completion
  4. systemd defaults: CPUWeight=300, IOWeight=300, Nice=-5, HEARTBEAT_INTERVAL=120

If your agent is older than PR #55:

BASH
gritivactl version
# if < 2.5.0
curl -fsSL https://get.gritiva.com/agent | sudo bash -s -- --upgrade

When to escalate#

  • If the loop persists after upgrade AND CPU pressure is normal — file a GitHub issue with the py-spy dump output and journalctl -u gritiva-agent --since '1 hour ago'
  • If multiple VMs simultaneously enter a loop — paging on-call (this is SEV-2 fleet-wide)