Runbook: Redis Memory Pressure

Last updated 2 min read
Report issue

Runbook: Redis Memory Pressure#

When you see: Redis OOM-killed, maxmemory-policy evicting keys, or Celery jobs failing with ConnectionError to Redis.

Severity: SEV-2 — Celery (background jobs, including DRM reconciliation) stops working.

Symptoms#

  • redis-cli info memory shows used_memory_rss / maxmemory > 0.9
  • Celery worker logs: ConnectionError: Error 32 connecting to redis: Broken pipe
  • DRM reconciler stops applying domain changes
  • Docs system pipeline stops processing new jobs

Diagnose#

BASH
# Inspect memory usage
redis-cli info memory | grep -E 'used_memory_human|maxmemory_human|maxmemory_policy'

# Show top-10 largest keys (sample)
redis-cli --bigkeys

# Show key-pattern memory usage
redis-cli --memkeys --memkeys-samples 1000

Common culprits:

  • Celery result backend keeping completed task results forever
  • A misbehaving cache writing without TTL
  • Pub/sub channels with stuck subscribers
  • A Lua script gone wrong storing huge structures

Mitigate#

Option A — Evict gracefully#

Set a sane eviction policy if you haven't:

BASH
redis-cli config set maxmemory-policy allkeys-lru
redis-cli config rewrite

This evicts least-recently-used keys when memory pressure hits.

Option B — Restart Redis (loses data)#

BASH
sudo systemctl restart redis

If you're using Redis only as a Celery broker / cache — losing the data is fine. If you've stored anything durable in Redis (bad idea, but it happens), this is destructive.

Fix (durable)#

  1. Audit who's writing to Redis without TTL:
BASH
redis-cli scan 0 count 100 | head -50 | while read k; do
  echo "$k -> ttl=$(redis-cli ttl "$k")"
done

Anything with ttl=-1 is forever-stored. Add an expire somewhere upstream.

  1. Configure Celery to expire completed results:
PYTHON
# in celeryconfig.py
result_expires = 3600  # 1h
  1. Bump Redis maxmemory if you've genuinely outgrown it. The control plane needs ~100 MB; Celery another 200-500 MB depending on backlog.
BASH
redis-cli config set maxmemory 1gb
redis-cli config rewrite