Runbook: WebSocket Disconnect Storm

Last updated 2 min read
Report issue

Runbook: WebSocket Disconnect Storm#

When you see: many agents simultaneously dropping their WS connection to the panel and reconnecting in a tight loop.

Severity: SEV-2. Affects telemetry visibility but not the gmesh data plane.

Symptoms#

  • Panel shows >10% of fleet flipping between Online and Offline within a 5-minute window
  • api.gritiva.com access logs show a spike in WS upgrade requests
  • journalctl -u gritiva-agent on affected agents shows WebSocket disconnected: 1011
  • The data plane (gmesh) is unaffected — your apps keep running, telemetry just stops

Diagnose#

Step 1 — Is it the panel side?#

BASH
# On a healthy laptop:
curl -sI https://api.gritiva.com/healthz
# expect HTTP 200 within 1s

If 5xx or timeout: panel is degraded; check status.gritiva.com. Stop here — not your problem to fix.

Step 2 — Is it the agents' side?#

BASH
ssh <one-affected-vm>
sudo journalctl -u gritiva-agent --since "10 minutes ago" | grep -iE 'disconnect|1011|timeout|reset'

Look for: 1011 internal error, timeout reading from peer, connection reset by peer.

Step 3 — Is it the network in between?#

Common culprits:

  • Cloudflare WAF rule change (rare, but happened in 2026-04)
  • Customer's egress proxy / firewall changed
  • DNS cache poisoning at the customer's resolver
BASH
# From an affected VM:
mtr -rwbz4 api.gritiva.com

Mitigate#

The agent's reconnect logic uses exponential backoff capped at 60s. If the storm is sustained, the agent will settle into 60-second poll intervals — annoying but stable.

To force a clean reconnect on a single VM:

BASH
sudo systemctl restart gritiva-agent

To pause an agent (e.g. during scheduled maintenance):

BASH
sudo systemctl stop gritiva-agent
# don't forget to start it again afterwards

Fix (durable)#

If the cause was a panel-side issue: it'll be in the relevant postmortem.

If the cause was a network issue specific to your environment, the agent has a WS_RECONNECT_BACKOFF_MAX env var you can tune via the systemd unit:

BASH
sudo systemctl edit gritiva-agent
# [Service]
# Environment="WS_RECONNECT_BACKOFF_MAX=120"
sudo systemctl restart gritiva-agent

Larger value = less load on the panel during outages, slower recovery when things come back.