Runbook: Disaster Recovery Plan

Last updated May 14, 20263 min read

Runbook: Disaster Recovery Plan#

When the worst happens: a whole VM lost, a region offline, or the hosted control plane unreachable for a long stretch.

Severity: SEV-1 by definition.

What "disaster" means here#

Scenario	Severity	Plan
Single scope down	SEV-3	restart, see Common Errors
Single VM down (host issue)	SEV-2	new VM, restore from agent backup
Whole region down (cloud provider outage)	SEV-1	failover to another region (HA setup required)
Control plane down (panel.gritiva.com unreachable)	SEV-1	scopes keep running; new lifecycle ops queue
Mesh totally fragmented	SEV-1	manual peer reset, see Mesh stuck runbook

RTO / RPO targets#

Single VM loss — RTO 30 min (with a backup), RPO 24h (daily backup cadence)
Region loss — depends on your HA setup; we don't multi-region by default today
Control plane — we maintain 99.9% uptime; if down, no RTO contract for self-hosted

Procedure: lost a single VM#

This is the most likely scenario. Assumes you have agent-state backups (see Backup & Restore).

Step 1 — Provision a fresh Linux VM#

Whatever your usual provisioning workflow is. SSH in, sudo available.

Step 2 — Install the agent (any token will do; we'll restore the right one in step 3)#

BASH
curl -fsSL https://get.gritiva.com/agent | sudo bash
# (no token argument; it'll just install + wait)
sudo systemctl stop gritiva-agent

Step 3 — Restore the previous VM's agent state#

BASH
# Get your last backup
scp old-backup-host:/root/backups/gritiva-2026-05-14.tgz .
sudo tar xzf gritiva-2026-05-14.tgz -C /
sudo systemctl restart gritiva-agent

Step 4 — Verify#

BASH
sudo gritivactl status
# expect: connected: yes, scopes restored

Within ~60 seconds, the panel should show the VM back online. Scope contents (your application data inside) need to be restored separately — they're in your application backups.

Procedure: control plane unreachable#

If panel.gritiva.com is down, your existing scopes keep running on each VM (they're independent processes). What's NOT available:

Creating new scopes
Adding domains
Mesh peer changes
Telemetry/dashboards

You can do emergency operations locally on each VM:

BASH
sudo gritivactl --offline scope start <scope>
sudo gritivactl --offline scope stop <scope>

When the control plane comes back, the agent re-syncs and any drift between local state and desired state gets reconciled.

Procedure: gmesh totally fragmented#

Rare. Indicates a major bug or a network partition.

BASH
# On each VM:
sudo gmeshd reset --all-peers
sudo systemctl restart gritiva-agent

This re-fetches the entire peer roster from the control plane and re-installs WireGuard configs from scratch. Expect ~2 minutes of full mesh outage during this.

Tabletop exercises (recommended)#

Run a tabletop exercise quarterly. Suggested scenarios:

"VM prod-eu-1 is gone. Walk me through getting it back."
"panel.gritiva.com is unreachable. We need to start a scope. Show me."
"All gmesh peers between two VMs are stuck. How do you confirm and fix?"

Record actual time-to-recovery vs your RTO target. Adjust your runbooks based on what surprised you.