Runbook: Certificate Expiry

Last updated 2 min read
Report issue

Runbook: Certificate Expiry#

When you see: HTTPS errors, browsers warning about expired certs, monitoring alerts about cert expiry within 14 days.

Severity: SEV-3 if 14-7 days out; SEV-2 if <7 days; SEV-1 if already expired and customer-visible.

Architecture recap#

GritivaCore TLS lives in two places:

  • Edge (Cloudflare) — wildcard *.gritiva.com cert managed by Cloudflare. You do nothing. Auto-renewed.
  • Per-domain (DRM) — when you gritivactl domain add app.example.com, DRM either uses Cloudflare's edge cert OR provisions a Let's Encrypt cert inside the scope's nginx. The Let's Encrypt path is what can expire.

Diagnose#

Step 1 — Identify which domains are at risk#

BASH
sudo gritivactl domain list --certs
# shows: domain, scope, expires_in_days, issuer

Anything with expires_in_days < 14 or issuer=letsencrypt deserves attention.

Step 2 — Check the renewal cron#

BASH
sudo gritivactl scope exec <scope> -- systemctl status certbot.timer
# expect: active (waiting); Trigger: <future date>

If Failed or absent, certbot can't run.

Step 3 — Check the renewal logs#

BASH
sudo gritivactl scope exec <scope> -- journalctl -u certbot --since "7 days ago" | tail -50

Look for Failed authorization, connection refused, rate limited.

Mitigate#

Force a renewal now#

BASH
sudo gritivactl scope exec <scope> -- certbot renew --force-renewal

If the renewal fails for the same reason it's been failing, fix the underlying cause first (see below).

Switch to Cloudflare edge cert (avoids LE entirely)#

BASH
sudo gritivactl domain edit app.example.com --cert cloudflare-edge

This routes the domain through Cloudflare's wildcard cert — no per-domain provisioning.

Fix (durable)#

Common causes of LE renewal failure:

  1. Port 80 not reachable from public internet (LE HTTP-01 challenge needs it). For scopes behind cloudflared tunnels, switch to DNS-01 challenge.
  2. Rate limited — LE limits to 50 certs per registered domain per week. If you hit it, switch to CF edge cert temporarily.
  3. DNS misconfiguration — dig +short app.example.com should return a Cloudflare IP. If it's pointing at your VM directly, the routing is wrong.

Prevention#

The DRM reconciler emits an expires_soon event 14 days before expiry. Wire this to your alerting:

BASH
gritivactl alerts subscribe --event domain.expires_soon --webhook https://your-slack-incoming-url