AI Agent Consistency: IBM Research Tackles the Reliability Gap

👁 115 reads

IBM Research has published an analysis of AI agent repeatability on Hugging Face that reveals a fundamental crack between what benchmarks show and what production users actually need.

A ReAct agent based on GPT-4.1, tested on the AppWorld test_normal dataset, will get the correct answer on average 77.4% of the time. That’s the number you see on leaderboards. But if you repeat the same task five times and require the agent to pass all five attempts, the percentage drops to 53.0%. The gap is 24.4 percentage points. On the hardest tasks, it reaches 30 points.

The difference between these two metrics isn’t a technical curiosity. In production contexts - verifying commitments in a contract, reconciling financial transactions - a workflow that succeeds once but fails on an identical request the next time is a problem, not a statistic.

Two Metrics, Two Questions

The researchers highlight an important terminological distinction. Mean@k answers the question: how good is the agent on average? Pass^k (not to be confused with the popular Pass@k) answers the question real users ask: will the agent execute the same task correctly every time? Pass@k is optimistic - one success out of k attempts is enough. Pass^k is pessimistic - every attempt must succeed. Most benchmarks report only the first number.

Diagnosis and Treatment

IBM Research built the Consistency Analyzer - a tool that analyzes a recorded agent trajectory and searches for decision points where the model was one token away from choosing a different path. Importantly, the tool works on a single recorded trajectory and doesn’t require ground truth or re-running the entire task. At each decision point, it retrieves k=5 completions with one additional model call.

Based on this diagnosis, the ALTK-Evolve system generates consistency guidelines - injected into the agent at inference time. The result: the Pass^5 gap drops from 24.4 to 12.0 pp. For tasks of the same type, the Pass^5 score increases by 16.0 pp; for similar tasks, by 13.0 pp. Mean accuracy doesn’t drop.

The full methodology and results are available in the technical report on arXiv.

It’s an interesting reminder that improving mean accuracy by a few points may be less valuable than making an agent stop randomly varying its answer to the same question.

// read moreIBM Research Tool Calibrates AI Agent Memory RequirementsNewsIBM Research Method Cuts Agent Memory Tokens Without Dropping LessonsNewsInherent Claims 27B Agent Beat Claude, GPT-5.5 at ResearchNews

Kinetyra