AI Decision Models: Decision Index Benchmark

👁 146 reads

For a decision-model benchmark, I look at three things together: whether the selected answer is right, whether its confidence is calibrated, and what an accepted decision costs. A high probability is a model output, not a guarantee. The ranking and local experiments below are dated snapshots; they are not a new benchmark run.

Compare the decision you will actually automate

The Artificial Analysis launch methodology separates classification, tool selection, knowledge and reasoning. A model that routes support tickets well can still be poor at deciding whether an unfamiliar factual claim is true. I would choose the relevant evaluation before comparing the overall score.

For an internal trial, keep a held-out set of real cases, including ambiguous examples that should go to a person. Freeze the labels and instructions, then record wrong automatic actions as well as correct ones. Raising a confidence threshold can reduce errors while sending more work to review. That trade-off belongs in the result, alongside latency and the bill.

Decision model benchmark: what should I compare?

Compare the same task, dataset, model version and decision threshold. Keep USD per task separate from USD per million tokens: the conversion depends on the size of your inputs. The historical price table below is not a current quote for a production deployment.

Related guides: AI model benchmarks · Claude guide · Coding agents

Sources checked on 9 November 2026

• https://artificialanalysis.ai/articles/artificial-analysis-decision-index

Earlier comparison and examples

A decision model is an AI model that doesn’t write text - it chooses an answer from a list you provide and tells you how confident it is. Instead of “this ticket is probably about payments,” you get payments: 0.97, and your code decides what to do with it. On October 1, 2026, Cloudflare released open models Clef and Clef-flash, and AWS released Strands Decider 2B the same day. We tested both on our RTX 4090 with 30 customer tickets in Polish.

What Is a Decision Model

A regular language model (LLM) generates text word by word. Even if you tell it to answer with one word or JSON, it’s still writing, just shorter. A decision model works differently:

1. It receives state - ticket text, order data JSON, and in Clef’s case also an image.
2. It receives questions with a closed list of answers: “which department?” with options payments, delivery, return…
3. It processes the state once and for each allowed option returns a probability. No generation, no parsing, no chance it will answer with an option outside the list.

All models in this family use three types of questions, popularized by Jev from TypeSafe:

The most important practical difference: the number you get is a probability calculated by the model, not text like “I’m 90% sure,” which an LLM might write based on intuition. This lets you set a simple threshold: above 0.8 the system acts automatically, below that the ticket goes to a human.

What It’s Good For (and What It’s Not)

Good use cases:

• Ticket routing - which department, is it urgent, what language, is the customer threatening to leave.
• Moderation - does the comment violate rules, is it spam, is the image suitable for publication (Clef supports images).
• Document classification - invoice type, expense category, does the contract contain clause X.
• Tool selection in agents - instead of asking a large model “which function to call?”, you ask a decision model to choose from a list of tools (more about the mechanism itself in the text on function calling).
• Gates in agents - does the agent’s response meet criteria, should it escalate to a human, which model should handle the query.

Bad use cases:

• writing, summarizing, translating, chat, code - a decision model won’t generate a single sentence,
• open-ended questions (“what does the customer mean?”) - if you don’t provide a list of options, there’s nothing to choose from,
• difficult multi-step reasoning - Strands Decider authors write explicitly that the model is clearly weaker than reasoning models in complex problems.

In practice, a decision model sits alongside an LLM: it makes hundreds of small, fast decisions, and the large model writes a response where text is needed. How to build an agent where such elements combine is shown in the guide AI Agents from Scratch.

Decision Models in October 2026

All open models in the table are compatible with the System One API, meaning Jev’s request format (POST /v1/systemone, state and questions fields). The same client code will work with any of them - you just change the address and model name. I wrote about OpenAI’s Decisions API when covering Agents API.

Cloudflare Clef and Clef-flash

Clef takes a frozen Qwen model, passes input through it once (only the so-called prefill, without generation) and feeds hidden states to a small network that Cloudflare calls a joint schema head. This head evaluates all options of all questions at once and can compare questions with each other. According to Cloudflare, this is the first model in this family with an image encoder and a 64k token window. I described the launch details and benchmark results in the news Cloudflare Clef: open decision model, and Cloudflare as a company in the guide Cloudflare.

Amazon Strands Decider 2B

The smallest of the open models. AWS kept only the “trunk” from Qwen3.5-2B-Base, removed the language head and added a pointer head with just over a million parameters. Plus a LoRA adapter of rank 16. On Hugging Face there’s a checkpoint strands-decider-2B-hobson-v19, alongside training data and scripts. According to the authors, median latency is about 115 ms on RTX 3090 and 153 ms on MacBook with M3. Result on public JevBench: 167 out of 231 tasks (72.3%).

Kev 1.0

Kev is a project by Jared Palmer (creator of Formik and Turborepo, among others): four sizes from 0.8B to 27B, LoRA and pointer head on Qwen3.5/Qwen3.8, Apache-2.0 license. It implements the public System One API, so TypeSafe SDK works with a local Kev server without changes. The model is trained and validated only in English, and the 9B version according to the model card takes about 22 GB of card memory. Kev didn’t make it into our test - I explain why below.

Pricing and Hardware Requirements

Workers AI has a free allocation of 10,000 “neurons” daily in Free and Paid plans. Clef costs 21,818 neurons per million input tokens, and Clef-flash 8,182, so you get about 458k tokens for Clef and about 1.2 million for Clef-flash daily for free. A short ticket with two questions is about 150-250 tokens for us, meaning several thousand decisions daily for free. You need a Cloudflare account - I didn’t set one up for the test, so I only know the hosted version from documentation.

Our Test: 30 Polish Tickets on RTX 4090

We wrote 30 tickets typical for an online store or SaaS company: payments, delivery, returns, account, outages and sales, 5 per department. Some deliberately without Polish diacritics, with typos or on the border of two departments (e.g. “still no refund” - is that returns or payments?). Each model got the same task: department (6 options) and whether the matter requires action today. We tested decision models in Polish with their native API, and LLMs through Ollama or OpenRouter with enforced JSON schema and temperature 0.

What this shows:

• Department assignment was too easy to determine ranking. 6 out of 8 models got 30/30. With 30 tickets, a one-error difference is noise. Only Llama 3.1 8B performed poorly (25/30) - it confused invoices with sales and coffee machine complaints with outages.
• Speed is a real advantage. Clef-flash on the same card responded 3.7 times faster than Gemma 4 12B and almost 7 times faster than Qwen3.8 27B, which has nearly identical base to large Clef. And that’s without optimized kernels (flash-linear-attention and causal-conv1d don’t install easily on Windows, so transformers library computed linear attention layers in slower reference version).
• Threshold must be tuned on your data. Strands Decider in Polish underestimated urgency probabilities: no ticket exceeded 0.5, so at default threshold it marked nothing as urgent. The order was sensible though, and at 0.4 threshold the result rose to 26/30. Clef-flash was opposite: at 0.5 it gave 4 false alarms, at 0.95 only 2 errors.
• Polish works, though models were trained mainly in English. The same tickets translated to English didn’t give better results: Clef-flash had 29/30, Strands 29/30 (both mistook a plan change during billing period).
• Urgency is subjective. We marked 6 tickets as urgent (502 error, account hack, double charge…). Almost all models also considered urgent a BLIK payment that didn’t register in the panel. You could agree with them.

Test cost: 0 USD for local models and about 0.04 USD on OpenRouter for 180 queries to Mistral and Gemini. Gemini 3.8 Flash doesn’t allow disabling reasoning, so it ran at lowest level, hence the ~1.8s time. We have scripts and tickets locally and will repeat the test with next versions.

Why We Didn’t Test Large Clef and Kev

• Clef 27B weighs 55 GB in bf16. Won’t fit on 24 GB without quantization, and Cloudflare hasn’t released a quantized version. The model has its own head and custom code (joint_schema_model.py), so it won’t run in Ollama or llama.cpp from GGUF file. Theoretically you can load it in 4 bits through bitsandbytes, but that would be a different model than what Cloudflare evaluates.
• Kev-9B needs a separate Qwen3.5-9B-Base base (another ~19 GB download), and we set a 40 GB limit for this test. Besides, the author declares English only.
• Workers AI requires a Cloudflare account, which we didn’t set up for the test.

Decision Model vs LLM with Structured Output

The same task can be done with a regular LLM that you force to return JSON conforming to a schema (I described this in the text Structured output: how to force AI to respond in JSON). When to choose what:

Our rule: if the answer is from a closed list and there are many decisions (hundreds daily and more), we start with a decision model. If we need justification or the answer depends on many reasoning steps - LLM. Best works a combination: decision model handles certain cases, and uncertain ones (below threshold) go to a large model or human.

How to Start in 10 Minutes

1. Install Python 3.10+ and PyTorch with CUDA (without NVIDIA card it will work too, just slower).
2. pip install strands-decider - this is the smallest model, about 4.4 GB to download.
3. Ask your first question from command line:

   strands-decider ask StrandsAgents/strands-decider-2B-hobson-v19 
     --state "You charged me twice for the same subscription" 
     --choice "Which department?=payments,delivery,return,account,outage,sales" 
     --noul "Is the matter urgent?"

4. When it works, start HTTP server (strands-decider serve ... --port 8000) and send requests in System One format. Server listens only on 127.0.0.1 and has no authentication - add your own layer for production.
5. If you have 24 GB VRAM, switch to Clef-flash: snapshot_download("Cloudflare/clef-flash") and systemone() function from included file. Request format stays the same.

If you don’t have your own server with GPU, there’s Workers AI or TypeSafe API. You can also run a model locally on a regular computer through Ollama, but only classic LLMs - decision models have custom heads and need Python with transformers.

Who Is This For

• Yes: stores and companies with helpdesk that want to automatically route tickets; teams building agents with many small decisions; content and image moderation; anyone who today pays for a large model just to answer “yes” or “no”.
• No: if you need text, conversation or code; if there are a few decisions daily (then convenience of large model wins); if you don’t have dozens of labeled examples to set thresholds on.

Frequently Asked Questions

What is an AI decision model?

It’s a model that instead of generating text chooses an answer from a provided list and returns probability for each option. It supports three question types: choice from list (choice), yes/no (noul) and rating on scale (score).

What is Cloudflare Clef?

A family of open decision models from Cloudflare from October 1, 2026: Clef (27B, on Qwen3.8-27B) and Clef-flash (9B, on Qwen3.5-9B). Apache-2.0 license, weights on Hugging Face, hosting in Workers AI. They support text, JSON, images and video, 64k token window.

Is Clef free?

Weights yes (Apache-2.0, commercially too). In Workers AI you pay 0.24 USD (Clef) or 0.09 USD (Clef-flash) per million input tokens, with free allocation of 10,000 neurons daily, giving about 458k or 1.2 million tokens.

Do decision models work in Polish?

In our test yes: Clef-flash and Strands Decider 2B assigned department correctly in 30 out of 30 Polish tickets. Weaker with calibration - probabilities in Polish were sometimes shifted, so threshold must be set on your own examples. Kev declares English only.

What computer is needed to run Clef locally?

Clef-flash needed 18.3 GB card memory for us, so RTX 3090, 4090 or 5090 is enough. Full Clef (55 GB in bf16) requires H100/H200 class card or quantization, which officially doesn’t exist. Strands Decider 2B fits in under 4 GB and works on CPU and Mac too. More about memory in the text how much VRAM is needed.

How does a decision model differ from function calling?

Function calling is how an LLM generates a function call with arguments. A decision model can do the part that involves choosing a tool from a list, but won’t generate text arguments. In agents both are often used: decision model chooses, LLM fills in details.

Can Strands Decider be used commercially?

Yes. Model, code and training data are under Apache-2.0 license, and base model Qwen3.5-2B-Base is also Apache-2.0. Included server has no authentication, so in production you need to hide it behind your own gateway.

What is Jev and System One?

Jev is a decision model from TypeSafe, released September 15, 2026 via API, and System One is the format of that API (state + named questions + typed responses). Clef, Kev and Strands Decider are compatible with it. More in the text TypeSafe and Jev model.

Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →

Sources

• Cloudflare: Introducing Clef (1.10.2026)
• Cloudflare/clef and Cloudflare/clef-flash on Hugging Face
• Workers AI pricing
• Strands Agents: Introducing Strands Decider 2B (1.10.2026) and model card
• Kev on GitHub and jaredpalmer/kev-9b
• Own test: October 5, 2026, RTX 4090, Windows 11, PyTorch 2.11, transformers 5.18, Ollama.

Verified: October 5, 2026.

Kinetyra