A Watchdog That Knows When Not to Heal

Flow

Adopting an intent-aware healer

1. 1Deploy everywhere in observe⌄

   Same logic, decisions only logged and counted as would-actions.

2. 2Audit the would-counters⌄

   Compare against planned shutdowns and known incidents.

3. 3Promote one low-stakes site⌄

   Switch to act mode where a mistake costs least.

4. 4Promote the rest⌄

   Flip remaining hosts once their rehearsal looks right.

5. 5Keep the backstop⌄

   An independent guest-down alert stays on regardless of the healer.

You want guests that crash to come back on their own, and you do not want to be woken up for a guest that just needed a restart. You also do not want a watchdog that fights your maintenance, resumes a guest into a full disk, or hides a crash loop. Here is the applied path one small fleet took when it replaced several hand-rolled guardian scripts with a single reconciler per hypervisor.

1. Pick the source of truth

Use the flag your platform already has. If a guest is set to start on boot, it is supposed to be running. Add one opt-out tag for exceptions. Do not keep a separate allowlist inside the watchdog: it drifts, and new guests silently fall outside it.

2. Write the decision as a pure function

Keep the "what should happen" logic separate from the commands that make it happen. That makes observe mode free and makes the logic testable.

from dataclasses import dataclass

DELIBERATE = {"stop", "shutdown", "pause", "suspend"}

@dataclass
class Guest:
    status: str            # running | stopped | paused
    run_state: str         # finer state, e.g. io-error
    locked: bool
    last_task: str
    seen_stopped_before: bool
    agent_fail_runs: int
    agent_ever_seen: bool
    no_reset: bool

def decide(g: Guest, agent_fail_threshold: int) -> tuple[str, str]:
    if g.status == "running" and g.run_state == "running":
        if g.agent_ever_seen and g.agent_fail_runs >= agent_fail_threshold:
            if g.no_reset or g.locked:
                return ("block", "agent-failing-no-reset")
            return ("reset", "agent-dead")
        return ("none", "healthy")
    if g.locked:
        return ("block", "locked")
    if g.run_state == "io-error":
        return ("block", "io-error")
    if g.last_task in DELIBERATE:
        return ("block", "deliberate-" + g.last_task)
    if g.status == "stopped":
        if not g.seen_stopped_before:
            return ("block", "stopped-confirming")
        return ("start", "stopped")
    if g.status == "paused":
        return ("resume", "paused")
    return ("block", "unexpected-state")

Every branch returns a reason. "block" is not an error: it is a decision you will publish.

3. Gate every action through one function

The executor checks a per-guest sliding window before doing anything. In observe mode it only counts "would-start". In act mode it runs the command under a timeout and records success or failure separately.

4. Schedule it safely

# example-healer.timer
[Timer]
OnBootSec=5min
OnUnitActiveSec=1min
AccuracySec=10s

[Install]
WantedBy=timers.target

Pair it with a oneshot service ordered after the platform's own start-all-guests unit, wrap the script body in a non-blocking file lock so overlapping runs exit immediately, and skip the whole pass while the host is still inside a boot grace period.

5. Publish decisions, not just actions

Write a metrics file each pass, to a temp name and then rename it so a scraper never reads half a file. Include: last run time, mode, skip reason, a gauge per blocked guest labelled with its reason, and a counter per action.

6. Roll out in observe mode

Deploy to every host in observe mode. Read the would-counters for a while and compare them with what you know happened: did it want to restart the guest you shut down on purpose? If yes, your deliberate-stop detection is wrong. Promote one low-stakes site to act mode first, then the rest.

7. Keep a separate backstop

Do not let the healer be its own only monitor. A plain alert that fires when a start-on-boot guest has been down for several minutes, joined against the same intent flag, works even if the healer is broken or inhibited.

How to verify it before trusting it

Rehearse each refusal on a throwaway guest while the host is in observe mode, and check that the published reason matches what you did:

• Shut the guest down from the platform's own tools. The next passes should report it as deliberate, never as "would start."
• Kill the guest's process from the host so it stops without a recorded shutdown task. After two passes the counters should show one would-start.
• Take a snapshot or backup of it and watch for a locked reason while the operation runs.
• Create the maintenance inhibit and confirm the whole pass reports a skip reason, then remove it.
• Stop the timer and confirm the staleness alert fires; a healer you cannot see is a healer you cannot trust.

Only when every one of those reads correctly is the host ready for act mode.

Tradeoffs you are choosing

The confirmation pass adds a minute to recovery. The action budget means a genuinely flapping guest stays down until a person looks. Resets of hung guests are blunt; that is why routers and storage are excluded and covered by network probes instead. These are deliberate costs, and each one is visible.