Batch Processing for Agent Training and Evaluation

Hermes batch processing runs many independent prompts through full agent sessions and records structured trajectories. It is primarily a research and engineering tool for training-data generation, model evaluation, and tool-use analysis—not a replacement for ordinary cron jobs or business queues.

Official documentation: https://hermes-agent.nousresearch.com/docs/user-guide/features/batch-processing

Input and output shape

The batch runner reads JSONL with one prompt object per line. Entries can also identify an isolated working directory, image, or container environment when the evaluation needs it. Each sample produces conversation history, tool-call statistics, reasoning coverage data, and checkpointed run output.

Choose the model, maximum turns, worker count, toolset distribution, and sample limit deliberately. Resume support is useful for interrupted runs, but it should be tested before launching an expensive dataset.

Batch-release checklist

• Validate every JSONL row before launch.
• Remove secrets, personal data, and copyrighted private corpora.
• Start with a tiny sample limit.
• Pin the model and provider route.
• Bound maximum turns and output tokens.
• Match worker concurrency to provider and machine limits.
• Use isolated containers or workdirs for prompts that run code.
• Record the dataset version, run name, configuration, and expected output.

Evaluation discipline

A successful process exit does not prove the trajectories are useful. Sample outputs across prompt classes, inspect tool selection, check failure rates, and compare statistics with the intended benchmark. Separate generation data from the independent evaluation set.

Pitfalls

• Sending thousands of prompts before validating ten.
• Using production secrets or customer records as convenient examples.
• Comparing models with different tools, limits, or provider routing.
• Increasing concurrency until rate limits distort the evaluation.
• Treating resumption as correct without checking duplicate or skipped samples.
• Fine-tuning on trajectories that were never inspected for unsafe actions.

Verification steps

1. Validate the dataset schema and count lines.
2. Run two or three harmless samples with one worker.
3. Inspect each trajectory and its tool-call statistics.
4. Interrupt a test run, resume it, and verify sample accounting.
5. Increase concurrency gradually while watching provider and system errors.
6. Check output statistics and manually review a representative sample.
7. Archive the exact run configuration with the evaluation report.