Concepts: Guardians

Last updated 3 min read
Report issue

Guardians#

A Guardian watches one target and decides what to do when it looks unhealthy. It is the answer to "who restarts the service at 3am" — with the important detail that it can only take actions you have explicitly allowed, and only ones that are reversible.

Each Guardian is a policy (what to watch, how carefully, what it may do) that produces incidents (what it saw, what it decided, what happened next).

Manage them in the panel under GUARDIANS.

What a Guardian watches#

A policy targets one of: a vm, a scope, a service, a network or a project.

It runs a set of checks on a fixed cadence (60 seconds by default):

checkwhat it looks at
healththe target's reported health state
logsrecent log output for error signatures
metricsCPU, memory and I/O against the target's own baseline
anomalydeviation from that baseline rather than a fixed threshold
alertsalerts already raised by the platform

Sensitivity (low, normal, high) tunes how eagerly those checks fire.

Care levels#

The level decides how far a Guardian may go on its own. They escalate:

levelbehaviour
observerecords what it sees; never notifies
notifytells you something is wrong
suggesttells you, and proposes a specific fix
auto_healapplies a fix from its allow-list, then tells you
auto_heal_escalateapplies a fix and escalates if the problem persists

Two safety properties hold at every level.

Auto-heal is reversible-only. The actions a policy may take live in an explicit allow-list (restart_service, scope_action:restart and similar). Destructive operations are never stored there, so a Guardian cannot delete a Scope, drop data or tear down a network no matter how it reads the evidence. An empty allow-list downgrades the policy to suggest-only even if the level says auto-heal.

Actions are rate-capped. max_actions_per_hour (3 by default) bounds how often a Guardian may act. A policy that keeps failing trips a circuit breaker and stops acting until you look at it — a restart loop is not made faster by restarting faster.

Incidents#

Every detection opens an incident carrying the evidence snapshot, the verdict, the action taken and the outcome. A decision is one of observed, notified, suggested, healed, escalated or failed, and an incident has a severity and a resolution time.

Incidents deduplicate. A persistent problem produces one incident with a cooldown rather than a new alert every tick — the same approach the platform's own alerting uses, and the reason a broken service does not generate a hundred notifications overnight.

Read them in the panel under GUARDIANS, or over the API:

BASH
curl -H "X-API-Key: $GRITIVA_API_KEY" \
  https://api.gritiva.com/api/v1/guardian/incidents

GET /api/v1/guardian/policies lists your policies. GET /api/v1/guardian/incidents/stream is a Server-Sent Events endpoint for live updates — a client that waits for a complete response body will hang on it.

Notifications and quiet hours#

A policy notifies through in_app, email or telegram channels.

Quiet hours and maintenance windows suppress notification during periods you choose — useful when you are deploying and expect a service to bounce. Suppression affects notification, not observation: incidents are still recorded, so the history stays complete.

Getting started#

Start conservative and promote once you trust it:

  1. Create a policy at observe on one service you know well, and let it run a day.
  2. Read its incidents. If the things it noticed are the things you would have noticed, move it to notify, then suggest.
  3. Only then move to auto_heal, and give it exactly one action — usually restart_service.

A Guardian that fires on noise is worse than none, and sensitivity is easier to tune while it is still only watching.

See also Concepts: Audit Logs for how Guardian actions appear in the audit trail, and Monitoring Setup for the metrics they read.