HeyRocky

Tune Hermes Provider Request and Stale Timeouts

Hermes exposes provider-wide and model-specific controls for total request duration and non-streaming stale detection.

providers:
  example:
    request_timeout_seconds: 1800
    stale_timeout_seconds: 90
    models:
      large-model:
        timeout_seconds: 2400
        stale_timeout_seconds: 180

The model-level value overrides the provider value. These controls apply across primary and fallback clients and after credential rotation. For OpenAI-wire clients, the configured request timeout also wins over the legacy HERMES_API_TIMEOUT value.

Distinguish timeout layers

  • Request timeout bounds the overall provider call.
  • Non-stream stale timeout detects a call that produces no response for too long.
  • Socket read timeout bounds the wait for the next streamed chunk.
  • Stream stale detection catches keep-alive traffic with no real generated content.

A local model can spend minutes on prefill before its first token. Hermes raises or disables some implicit stale detectors for recognized local endpoints. An explicit value is still honored, so copying aggressive cloud settings onto a local model can create false failures.

The documented legacy defaults include a 1,800-second general API timeout, a 90-second non-stream stale detector, and 900 seconds for native Anthropic. AWS Bedrock uses its SDK timeout path and is not covered by these provider fields in the same way.

Tune from measurements

  1. Record time to first token and total completion time for representative prompts.
  2. Separate a slow healthy model from a dead connection.
  3. Set the provider baseline for normal traffic.
  4. Add model overrides only for real outliers.
  5. Recheck fallback behavior after changing the primary timeout.
  6. Leave local auto-adjustment implicit unless measurements justify an override.

Checklist

  • The transport and provider type are identified.
  • Streaming and non-streaming behavior are not conflated.
  • Local prefill latency is measured.
  • Model overrides are narrower than provider defaults.
  • Fallback timing matches the operational objective.
  • Bedrock is configured through its supported SDK path.

Common pitfalls

  • Lowering stale detection to make failover feel fast. Healthy long-prefill calls may be killed.
  • Raising every timeout after a hang. A dead call becomes slower to recover.
  • Assuming one timeout covers every transport. Streaming has separate read and stale layers.
  • Ignoring fallback cost. Faster handoff can move traffic to a more expensive route.

Verification

Run representative short and long prompts while recording time to first token, total duration, provider chosen, and fallback events. Test one controlled timeout in a non-production profile. Confirm the error is classified correctly and that the fallback route behaves as configured.

Official reference: Hermes provider timeout configuration.

Related pages