Context Compression and Prompt Caching

Long sessions need two different controls: compression reduces conversation history when context grows, while prompt caching reduces the cost of repeatedly sending a stable prefix. They interact, but they solve different problems.

Official documentation: https://hermes-agent.nousresearch.com/docs/developer-guide/context-compression-and-caching

Two compression layers

The in-loop context engine is the primary compression path. It protects the first exchange and a recent tail, prunes large old tool outputs, summarizes the middle, and keeps tool-call groups intact. Gateway session hygiene is a later safety net for messaging sessions that approach the context limit before a normal agent turn can recover.

The summary model must be capable of receiving the content that needs compression. If an auxiliary compression route has a smaller usable context than the main route, summary generation can fail and continuity can degrade. Treat auxiliary routing as part of reliability, not only cost control.

Prompt caching

Provider prompt caching rewards a byte-stable prefix. Stable identity, tool guidance, and project context should not be mutated casually mid-session. Model, provider-account, or credential rotation can invalidate a provider-side cache even when the visible conversation is unchanged.

Reliability checklist

• Keep compression enabled unless a measured use case requires otherwise.
• Protect enough recent messages for the active workflow.
• Route compression to a model with an adequate context window.
• Keep tool-call and tool-result groups together.
• Keep stable prompt layers deterministic.
• Expect cache misses after model or credential changes.
• Test long gateway sessions as well as CLI sessions.
• Inspect the compressed summary for decisions, files, blockers, and next steps.

Pitfalls

• Confusing a cache hit with retained conversational meaning.
• Sending oversized old tool output into every future turn.
• Choosing the cheapest summary model without checking its context limit.
• Mutating the system prompt between ordinary turns and destroying cache stability.
• Assuming compression cannot affect session lineage or search behavior.

Verification steps

1. Create a disposable session with repeated tool output and explicit decisions.
2. Drive it past the configured compression threshold.
3. Confirm old tool output is reduced and recent turns remain intact.
4. Inspect the structured summary for goals, constraints, progress, and files.
5. Continue the task and verify the agent follows the preserved decisions.
6. Check cache metrics before and after a stable follow-up.
7. Repeat after a deliberate model switch and document the expected cache reset.