Recover Hermes Kanban Tasks with Retries and Circuit Breakers

Hermes Kanban is a durable work queue, not a one-shot subagent call. A task remains the logical unit of work while each execution attempt becomes a separate run. That distinction makes recovery practical: operators can inspect why an attempt failed, preserve its evidence, repair the cause, and let a new worker continue on the same card.

The dispatcher normally runs inside the gateway and checks the board every 60 seconds. It reclaims dead or stale workers, promotes dependency-ready cards, claims work atomically, and applies a bounded circuit breaker so a broken card cannot respawn forever.

Understand the recovery states

A failed-looking card can represent several different conditions:

spawn_failed means the worker could not start, such as when an assignee is missing or a workspace cannot be mounted. It increments the consecutive-failure counter and returns the task to ready for another attempt.
crashed means the spawned worker process disappeared. The dispatcher releases the claim and makes the task eligible for a fresh worker.
timed_out means the task exceeded its configured runtime. Hermes terminates the process and requeues the card.
stale means a long-running task passed kanban.dispatch_stale_timeout_seconds—four hours by default—and had no heartbeat during the last hour. It returns to ready without increasing the failure counter.
protocol_violation means a worker exited successfully while the card was still running, usually because it never called kanban_complete or kanban_block.
gave_up means the effective failure limit was reached. The circuit breaker moves the task to blocked and records the last error.

Retries do not erase earlier work. task_runs keeps each attempt's outcome, error, summary, metadata, and timing, and the next worker receives prior attempts in kanban_show() context.

Inspect before changing state

Use the human-facing CLI to establish what happened:

hermes kanban show <task-id>
hermes kanban runs <task-id>
hermes kanban tail <task-id>
hermes kanban diagnostics --json

show gives the current card, comments, links, and recent state. runs separates attempts so you can see whether the pattern is a repeatable setup fault or a one-off crash. tail shows the event sequence, including spawn_failed, protocol_violation, and gave_up. diagnostics gives a board health snapshot.

Do not treat every blocked card as transient. Missing credentials, an unknown profile, an invalid workspace, or repeated worker-protocol failures will usually recur until the cause is corrected. Quota, authentication, and HTTP 429 failures may also trigger the respawn guard; waiting for the rate window or repairing authentication is more useful than forcing another immediate attempt.

Set a deliberate retry budget

The default dispatcher circuit-breaker limit for ordinary consecutive failures is kanban.failure_limit: 2. A task can override it at creation time:

hermes kanban create "Rebuild the search index" \
  --assignee ops \
  --max-retries 3

--max-retries is the number of non-successful attempts allowed before blocking, not the number of extra attempts after the first. A value of 1 blocks on the first failure. A value of 3 permits two retries and blocks on the third failure.

For a board-wide policy, edit config.yaml:

kanban:
  failure_limit: 2

The effective limit resolves from the task's max_retries, then the dispatcher's failure_limit or kanban.failure_limit, then the built-in default. Restart the gateway after changing dispatcher configuration so the running dispatcher reloads it.

Recover a task after the circuit trips

Repair the actual cause first. Then leave a concise durable note and unblock the card:

hermes kanban comment <task-id> "Fixed the worker dependency; retry with the same acceptance criteria."
hermes kanban unblock <task-id>
hermes kanban runs <task-id>

Unblocking moves the card to ready when all parents are done, or to todo while a parent remains open. The dispatcher will claim a ready card on a later tick. The new run receives the comment thread and prior run history, so record what changed rather than asking it to start blindly.

An unblock resets the dispatcher's consecutive-failure count, but it deliberately preserves repeated-block history. If the task is unblocked and then re-blocked for the same reason twice by default, the unblock-loop breaker routes it to triage for a human decision. Resolve the recurring dependency or capability gap instead of building an automatic unblock loop.

Keep long work recoverable

Workers should call kanban_heartbeat during long operations and at least once an hour when work may exceed an hour. A human can also record liveness from the CLI:

hermes kanban heartbeat <task-id> --note "Migration is still progressing in batches"

Set an explicit runtime ceiling when creating potentially unbounded work:

hermes kanban create "Audit the archive" --assignee ops --max-runtime 2h

Workers should finish through kanban_complete(summary=..., metadata=...) or stop through kanban_block(reason=...). Plain-text narration is not a lifecycle transition and can become a protocol violation.

Pitfalls

Do not confuse a retry budget with guaranteed success; it only bounds repeated failure.
Do not unblock before fixing a stable error. Repeated identical blocks can escalate the card to triage.
Do not have dispatcher-spawned workers shell out to hermes kanban. Workers use the task-scoped kanban_* tools; the CLI is for people, scripts, and cron.
Do not assume a stale event is a worker fault. It is absence detection and does not tick the failure counter.
Do not operate one SQLite board across multiple hosts. Hermes Kanban's dispatcher, PID checks, and board are intentionally single-host.

Verification checklist

Verify hermes kanban runs <task-id> shows every attempt and the expected terminal outcomes.
Verify the root cause named in the last failed run has actually been corrected.
Verify the task's --max-retries value or board-level kanban.failure_limit matches the intended risk budget.
Verify an unblocked task lands in ready, or in todo only because a parent is still open.
Verify the next worker adds heartbeats for long work and finishes with kanban_complete or kanban_block.
Verify the final run contains a useful summary, machine-readable evidence, and any remaining risk.