Runbook: Mesh Peer Stuck in CONNECTING

Last updated 2 min read
Report issue

Runbook: Mesh Peer Stuck in CONNECTING#

When you see: a gmesh peer in PEER_STATUS_CONNECTING for more than 5 minutes, with the mesh watchdog logging "manual intervention may be required."

Severity: SEV-3 (one peer pair affected) or SEV-2 (many peers, fleet-wide mesh degradation).

Symptoms#

  • gmeshctl peer list shows a peer with status=PEER_STATUS_CONNECTING and since >5 minutes
  • mesh_watchdog log lines every 60 seconds: update_peer requested for stuck peer X
  • Apps in the affected scopes can't reach each other across the mesh

Diagnose#

Step 1 — Confirm the peer state#

BASH
sudo gmeshctl peer list
# look for status=CONNECTING with since > 5min

Step 2 — Identify which side is broken#

The peer is (scope_id_A, scope_id_B). Check both:

BASH
# On VM hosting scope A:
sudo gritivactl scope exec scope-a -- ip link show wg-gmesh
# expect: <BROADCAST,MULTICAST,UP,LOWER_UP> ... wg-gmesh
# if "Device 'wg-gmesh' does not exist" — THAT side is broken

# Same on VM hosting scope B
ssh <other-vm>
sudo gritivactl scope exec scope-b -- ip link show wg-gmesh

The most common cause (per the 2026-05-14 postmortem for related issues): one scope is missing its wg-gmesh interface inside its netns. The peer roster is correct, but the local kernel state isn't.

Step 3 — Check WireGuard handshake state#

BASH
sudo gritivactl scope exec <scope> -- wg show wg-gmesh latest-handshakes
# look for the peer's pubkey + a recent timestamp
# if "0" or "1970…" — no handshake has happened

Mitigate#

Force-reconnect the affected peer:

BASH
sudo gritivactl scope mesh disconnect <scope> --peer <remote-vm>:<other-scope>
sudo gritivactl scope mesh connect <scope> --peer <remote-vm>:<other-scope> --force

This deletes the local peer entry, re-fetches it from the panel, and re-installs the WireGuard config.

If the issue persists after --force:

BASH
# Re-create the wg-gmesh interface entirely
sudo gritivactl scope mesh reset <scope>

This is more invasive — it tears down ALL peer connections for that scope and re-establishes them. Brief mesh outage for the scope (~10s).

Fix (durable)#

If you find that scopes 114 / 115 (or similar) keep showing this — they may be mesh-broken at the scope level (the scope's netns doesn't have a wg-gmesh interface to attach peers to). This was the cause of the 2026-05-14 5-day-stuck-peer issue:

BASH
sudo gmeshctl peer remove --id <peer_id>
sudo gritivactl scope destroy <scope>
sudo gritivactl scope create <scope> --cpu ... --mem ...   # fresh netns
sudo gritivactl scope mesh connect <scope> --peer ...

Action items still open#

The mesh_watchdog should auto-detect "remote scope has no wg-gmesh interface" and call scope_mesh_disconnect automatically instead of just logging "manual intervention may be required." Tracked at [GitHub #TBD].