Diagnose Agent Quality with a Baseline Test

When an agent suddenly feels less capable, avoid changing five settings at once. First create a small repeatable test that separates conversation state from model, tool, and configuration changes.

Define the symptom precisely

Replace “it feels dumber” with an observable failure:

it forgets an instruction stated earlier in the session;
it stops using a tool that is available;
it gives shallow answers to a previously successful prompt;
it ignores project context;
it repeats work after compression; or
it fails only in one platform, profile, or long-running session.

A precise symptom determines which evidence to collect.

Run the baseline sequence

Save the exact prompt and expected result.
Record the active profile, provider, model, and platform.
Use /usage to inspect token pressure and /context to see what occupies the context window.
Run the same prompt in a fresh session with the same model and toolset.
Compare the fresh result with the long-session result.
Change one variable at a time only after that comparison.

If the fresh session succeeds, the likely issue is context state, conflicting instructions, or accumulated conversation—not a universal model failure. If both sessions fail, inspect model routing, provider health, enabled tools, project context, and prompt clarity.

Recovery ladder

Remove ambiguity from the acceptance criteria.
Confirm required tools are actually enabled.
Check that the intended project/profile context loaded.
Use /compress when the conversation is valuable but context is crowded.
Use /new when a clean session is the better diagnostic.
Re-run the saved baseline prompt.

Exact context limits and defaults vary by model and release. Read the current runtime rather than relying on an old number.

Diagnostic checklist

One exact prompt is saved for comparison.
Expected output is concrete and testable.
Active model and provider are recorded.
Tool availability is verified.
Context usage is inspected.
Fresh and existing sessions are compared.
Only one variable changes per test.

Common pitfalls

Switching models before collecting evidence

A model switch can hide the original cause and creates a second variable.

Treating compression as a perfect transcript

Compression preserves important context in summarized form; it is not identical to retaining every message verbatim. Reconfirm exact constraints after compression when they matter.

Testing with a vague prompt

A broad request produces noisy comparisons. Use a short prompt with an objective pass condition.

Verification steps

Run the baseline in the original session.
Run it in a fresh session under the same routing.
Record the smallest observed difference.
Apply one recovery step.
Run the baseline again and confirm the targeted symptom changed.

Official reference

Follow the current Troubleshooting: My Agent Feels Dumber guide for supported diagnostics.