Runbook: Disk Full

Last updated 2 min read
Report issue

Runbook: Disk Full#

When you see: agent failing to write logs, scopes failing to start, journalctl errors about No space left on device.

Severity: SEV-2. Cascading failure — once disk is full, almost everything breaks.

Symptoms#

  • df -h shows 100% on / or /var
  • New scope creation fails with "no space left"
  • Existing scopes become read-only
  • journalctl -u gritiva-agent is empty (can't write)
  • Postgres / Redis on the same host start failing

Diagnose#

BASH
df -h | head -5
sudo du -sh /var/log/* 2>/dev/null | sort -h | tail -10
sudo du -sh /var/lib/* 2>/dev/null | sort -h | tail -10
sudo du -sh /var/lib/docker/* 2>/dev/null | sort -h | tail -10
# Check journal size specifically
sudo journalctl --disk-usage

Top usual suspects (in order of likelihood):

  1. Docker images / layers (/var/lib/docker/overlay2/*)
  2. Journal logs (/var/log/journal/*)
  3. Per-scope upper layers (/var/lib/gritiva/scopes/*/overlay/upper)
  4. Postgres WAL files
  5. Cores from a crashed process (/var/lib/systemd/coredump)

Mitigate (in order of safety)#

Step 1 — Trim journal logs (always safe)#

BASH
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-time=7d

Recovers anywhere from 100 MB to several GB.

Step 2 — Prune docker (safe if you're sure)#

BASH
sudo docker system prune -af --volumes

⚠️ This deletes ALL unused images, containers, networks, and volumes. If you have data in unnamed volumes, kiss it goodbye. Skip if unsure.

Step 3 — Prune scope overlay upper layers (advanced)#

BASH
# List big overlay uppers
sudo du -sh /var/lib/gritiva/scopes/*/overlay/upper | sort -h | tail -5
# If a specific scope is the offender, identify what's there
sudo du -sh /var/lib/gritiva/scopes/<id>/overlay/upper/*

This is per-scope writable filesystem. Don't blindly delete — your customer's data may be there.

Step 4 — Add a disk#

If you're genuinely at the host's natural limit, add disk and migrate. There is no clever fix.

Fix (durable)#

  1. Set log rotation:
BASH
sudo tee /etc/systemd/journald.conf.d/limit.conf <<EOF
[Journal]
SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=2week
EOF
sudo systemctl restart systemd-journald
  1. Set docker log rotation:
BASH
sudo tee /etc/docker/daemon.json <<EOF
{
  "log-driver": "json-file",
  "log-opts": {"max-size": "100m", "max-file": "3"}
}
EOF
sudo systemctl restart docker
  1. Monitor disk via the agent's metrics endpoint and alert at 80%.