Troubleshoot the Hermes Egress Proxy Without Weakening It

When an egress-protected Docker workflow fails, the fastest unsafe “fix” is to bypass the proxy or put real keys back in the sandbox. Resist that shortcut. Diagnose the path in layers: binary, state files, daemon, listener, Docker wiring, certificate trust, destination policy, token mapping, and upstream response.

Layered diagnosis

Start with hermes egress status. Separate “proxy is not running” from “proxy is running but the request failed.” A startup failure may come from a missing binary, unreadable state, occupied listener, missing Bitwarden value, or an invalid configuration. A request failure may instead be CA trust, proxy environment, denied destination, absent mapping, or the upstream provider.

Use the daemon log and audit log locally. Redact before sharing: proxy tokens, real keys, authorization headers, account identifiers, local paths, and private hosts are sensitive even when a request failed.

Troubleshooting checklist

Confirm Hermes recognizes the proxy as enabled and configured.
Confirm the pinned iron-proxy binary is present and reports a version.
Check that the host-side daemon is running exactly once.
Verify the configured listener is reachable from the Docker backend.
Confirm the CA certificate is mounted and CA-bundle variables are present.
Check that the requested hostname is allowed and not resolved into a denied range.
Confirm the provider has a mapping and the real host credential is available.
Distinguish TLS trust errors from upstream authentication or rate-limit errors.
Restart after setup or policy changes.
Preserve enforce_on_docker: true while diagnosing.

hermes egress status
hermes egress restart

Symptom map

Docker refuses to start: this can be correct fail-closed behavior. Restore proxy health instead of disabling enforcement.

Certificate verification fails: confirm the Hermes CA mount and runtime-specific CA variables. Node, Python, curl, and other clients can use different trust settings, so test the same runtime the workflow uses.

Allowed provider returns authentication failure: verify that the standard provider variable contains an opaque token in the sandbox and that the corresponding real value exists only on the host. Do not print either value.

One endpoint works and another is denied: inspect the actual destination hostname. Upload, OAuth, telemetry, and regional endpoints may differ from the primary API host; approve only the endpoint the workflow needs.

Status says running but requests time out: confirm the listener address and port selected for the platform, then inspect daemon logs for upstream resolution, policy, and TLS errors.

Common pitfalls

Disabling enforcement before identifying the failed layer.
Re-running setup repeatedly without restarting the in-memory daemon.
Testing with a different HTTP runtime than the production workflow.
Treating every 401 as a proxy failure; it can be an upstream account or credential issue.
Adding broad hosts to cure a DNS, TLS, or mapping problem.
Publishing environment dumps or verbose curl headers.
Killing processes by guessed PID instead of using the managed lifecycle commands.

Verification steps

Capture a redacted baseline from hermes egress status.
Run one minimal allowed request from the same Docker runtime and client library as production.
Confirm the audit record shows the expected allow/transform path.
Stop the proxy and verify Docker fails closed.
Start the proxy through Hermes, not a hand-built background command.
Repeat the allowed request and a denied-host fixture.
Restart the gateway or worker that owns the Docker backend and repeat once more to catch stale environment state.

Official documentation

https://hermes-agent.nousresearch.com/docs/user-guide/egress/iron-proxy