Decision Index v1.0 Released by Artificial Analysis
Artificial Analysis has launched Decision Index v1.0 to benchmark decision models against traditional LLMs on structured evaluation tasks.
👁 112 reads
On October 13, 2026, Artificial Analysis published the Decision Index v1.0, establishing the first independent benchmark for decision models. These systems evaluate structured prompts - covering multiple-choice selections, binary answers, and confidence scores - in a single request and yield explicit probabilities for every candidate option rather than generating tokenized text streams. The organization evaluated eleven decision models along with standard large language models queried via structured JSON outputs.
Benchmark Construction and Evaluation Weights
The Decision Index synthesizes four separate benchmarks, each weighted equally at 25 percent. AA-Omniscience-Decision presents 500 private knowledge questions with five options, including an explicit ‘I don’t know’ alternative, scoring models equally on basic accuracy and non-hallucination retention. HLE-MultiChoice tests complex text-only reasoning via 513 questions drawn from Humanity’s Last Exam. When2Call, developed by NVIDIA, measures whether an agent correctly answers directly, triggers a tool, requests clarification, or declines. Finally, Ai2’s WildGuardTest measures average F1 performance across harmful request detection, refusal tracking, and harmful response evaluation.
Top Leaderboard Results
According to Artificial Analysis figures released on October 13, 2026, Jev 1.13 from TypeSafe leads the benchmark, followed closely by GLiDE and Microsoft-Decision-1. While general-purpose LLMs achieve higher overall quality indexes, they do so with significantly higher latencies and running costs.
| Model | Index | Cost / 1M Tasks | Median Time |
|---|---|---|---|
| Jev 1.13 | 57.5 | $41.5 | 127 ms |
| GLiDE | 53.9 | $119.0 | 217 ms |
| Microsoft-Decision-1 | 53.8 | $30.7 | 207 ms |
| Decider v1.1 27B | 53.1 | $16.5 | 170 ms |
| Mercury Decide | 52.7 | $30.4 | 500 ms |
| Celeris-1 Decision | 52.3 | $31.6 | 63 ms |
| Claude Haiku 5.5 (low) | 60.6 | $184.0 | 1434 ms |
| Claude Opus 5.5 (max) | 79.0 | $51,295.0 | 3571 ms |
Across specific tasks, Artificial Analysis highlighted substantial operational variances. Jev 1.13 secured its overall lead primarily through reasoning, recording 20.9 on HLE-MultiChoice compared to 15.4 or less for all other decision models. In contrast, GLiDE and Celeris-1 Decision led tool routing on When2Call with scores of 83.7, outperforming Claude Haiku 5.5 low at 73 and GPT-6 Luna low at 77.
The benchmark reveals that general knowledge remains the core vulnerability of specialized decision architectures. Decision models posted knowledge accuracy between 28 percent and 41 percent on AA-Omniscience-Decision, trailing Claude Haiku 5.5 low at 52 percent and GPT-6 Luna low at 66 percent. While Claude Haiku 5.5 low scored 3.1 index points above Jev 1.13, Artificial Analysis noted that the LLM was 11 times slower and 4.4 times more expensive per task.
More: AI decision models - Decision Index ranking, prices and tests
More on this from QUANTSLANT
state of ai·IMF Warns of AI Crash as Deployments ExpandThe IMF warns of an AI market crash while real-world production deployments surge across defense, public infrastructure, and robotics.