Decision Models vs Small LLMs: Cost, Speed and Accuracy

👁 114 reads

In the Decision Index, decision models respond 3 to more than 20 times faster than small language models, but Claude Haiku 5.5 (low) scores higher on the overall index than any of them. Cost is less straightforward: it depends on the specific model and how many output tokens you pay for with an LLM. Here, I compare Artificial Analysis measurements, API pricing and a transparent token-based calculation.

Data current as of October 13, 2026. “Small language models” here means lower-cost options in providers’ product lines (Haiku, Luna, Flash and Gemma), not models with a confirmed small parameter count. For an explanation of decision models and the ranking, see the guide to AI decision models. A separate guide covers savings per million classifications and includes a cascade calculator: Using a decision model instead of an LLM. Here, I focus on comparing small LLMs with decision models.

Compare the same task first

A decision model returns a choice and the probability of each option. In the Decision Index, a standard language model gets the same tasks but must respond in a constrained JSON format. That means both types of model solve identical questions, even though they produce their answers differently (AA methodology).

So don’t compare one model’s Decision Index score with another model’s Intelligence Index score: they use different scales, even though both appear on the Artificial Analysis website. Also check the setting shown next to an LLM’s name: in AA, Haiku 5.5 in the low, medium, high, xhigh and max variants appears in five separate rows, each with a different score, cost and latency.

One more point: choosing the right category is not the same as answering a customer. If your process needs an explanation or a message after the decision, you need to add a text-generation step. Comparing a classifier alone with a complete text response changes what you’re measuring.

AA scores, cost and latency

Source: AA Decision Index ranking, live data accessed October 13, 2026, with language-model comparisons enabled. The table includes all 11 decision models in the ranking and selected small LLMs. Bold marks the best value in each column. The Gemini row indicates missing data, not a score of zero.

AA estimates cost from token usage and current prices, then calculates the average cost per task and scales it to one million tasks. Decision models pay only for input; LLMs pay for both input and output. Latency is the median for the index tasks. So this is not a price list for “a million arbitrary decisions”; it is a cost estimate for a specific set of questions.

What the table tells you:

• Haiku 5.5 (low) leads on score: it is 3.1 points ahead of Jev, but AA shows it as about 11 times slower and 4.4 times more expensive.
• Model category does not guarantee a lower price: Clef costs more than Haiku 5.5 (low) and GPT-6 Luna (low), although it responds faster. Clef-Omni costs more than Luna in the non-reasoning variant.
• A larger LLM does not mean better decision-making: Gemma 4 31B with reasoning is the most expensive and slowest model in the comparison, yet it only narrowly beats Luna (low) on score.
• Lowest-cost options: Decider v1.1 27B, Microsoft-Decision-1 and Jev 1.13 sit on AA’s cost-effectiveness frontier, offering the best score for their price.

Accuracy depends on the type of decision

The highest overall index score does not mean a model leads on every component. According to AA’s analysis, LLMs lead on knowledge, while leading decision models match or outperform them on safety classification and tool-use decisions.

Sources for the figures, accessed October 13, 2026: AA charts for AA-Omniscience-Decision, HLE-MultiChoice, When2Call and WildGuardTest. The final column collects the best scores from different models, so it does not describe a single system. For more on how the index is built and what each test measures, see my article on the Decision Index ranking.

Practical takeaway: if your task is to assign a message to a clearly defined category or choose an agent’s next step, a decision model is a serious contender. If the decision depends on knowledge that is not included in the prompt, a small LLM has a clear advantage. Treat this as guidance for choosing a test set, not a promise that you’ll get the same results at your company.

API pricing: a separate table, with separate units

Pricing sources, accessed October 13, 2026: Anthropic, OpenAI, Google and Cloudflare. Decision-model prices are from their providers, compiled in the guide. Haiku and Luna rates match the costs OpenRouter charged me for queries in my October 13 test. The table excludes prompt caching, batch discounts and surcharges for processing in the EU.

Note that Google lists Gemini 3.8 Flash rates of $0.75 and $3.75 as valid through the end of 2026, so check the current price before estimating costs for the following year. For every LLM, also check whether your input exceeds the long-context threshold, which triggers different rates.

What a million tasks costs under explicit assumptions

This is a scenario, not a benchmark. I assume one million calls, each with 1,000 billable input tokens. For LLMs, I compare 50 and 500 billable output tokens per task, including paid reasoning. These are assumed usage levels, not predictions of any model’s response length.

Cost of 1 million tasks in USD =
input tokens per task * price per 1 million input tokens
+ output tokens per task * price per 1 million output tokens

Haiku, with 50 output tokens:
1000 * 0.10 + 50 * 0.50 = 125 USD

My calculations use prices from October 13, 2026. Decision models do not generate text, so their costs do not depend on the output-token budget. Bold marks the lowest cost in each scenario, not the quality winner.

• With short output, Haiku and Luna cost less than Clef on Workers AI; with longer output, the comparison reverses.
• In this calculation, Gemini 3.8 Flash costs 7.5 times as much as Haiku and Luna under either output-token budget.
• If you need the extra output tokens for an explanation, a decision model cannot replace that. You’ll need another text-generating model, which means another call to include in your estimate.

Using the same 1,000-token input for every model is a simplification: different tokenizers count the same text differently. In my test, one Polish support ticket with questions used about 205 tokens in Mercury Decide and about 650 in Jev, while Haiku received an average of 782 input tokens with the instructions and JSON schema. For a real comparison, record the usage reported by the API rather than estimating it from message length.

Latency: time until the decision is ready

In a classification task, what matters is when your application can take the next step. The first character of a JSON response is not yet a decision, so time to first token and generation speed are no substitute for measuring the complete response.

My test on October 13, 2026, using queries from Poland through OpenRouter and therefore including network latency, supports the pattern in AA:

• Decision models: median latency of 310-619 ms, from the fastest, Clef-Omni, to the slowest, Mercury Decide.
• GPT-6 Luna (low): 1,371 ms.
• Claude Haiku 5.5 (low): 1,689 ms.
• Gemini 3.8 Flash: in a separate API test on October 5, 2026, median latency was 1,822 ms across 30 Polish support tickets (department classification: 30/30; urgency: 26/30). That was a different test set on a different day, so I have not added the figure to the AA table.

The median shows the middle of the distribution, not the slowest responses your users may have to wait for. In your own tests, also record the long tail, errors and retries. I calculated how many hours it takes to process a million decisions one after another, and what rate limits change, in the cost guide.

Accuracy in Polish: small LLMs still have a slight edge

On 47 unambiguous Polish support tickets in my test, decision models assigned the correct department in 41-44 cases, while Claude Haiku 5.5 (low) and GPT-6 Luna (low) did so in 45. A difference of one to three tickets in such a small sample is close to noise, but it is consistent with AA: small LLMs have a slight edge on simple classification, at the cost of speed and price. See the guide for full results across all 11 models.

Which option should you choose?

• Decision model: when the output is a label or choice of action and you have thousands of decisions to make. Test candidates on the same examples and use their probabilities to route uncertain cases for further review.
• Claude Haiku 5.5: when you need to combine the decision with text or knowledge beyond the prompt. The low variant has the highest score in this comparison, but responds more than 10 times slower than the fastest decision models.
• GPT-6 Luna: the highest accuracy on knowledge questions among the variants compared (66.0), but in this test it practically never chose “I don’t know” (0.0), so it guessed more often. Keep the low, non-reasoning and Decisions rows separate: they have different scores, costs and latencies.
• Gemini 3.8 Flash: you’ll need to run your own comparable test. Its price supports a scenario-based estimate, but does not fill the gap in the Decision Index.

Frequently asked questions

Is a decision model always cheaper than a small LLM?

No. In AA’s measurement, Clef costs $199.4 per million tasks, more than Claude Haiku 5.5 (low) and GPT-6 Luna (low). The price depends on the specific model, input length and paid output tokens.

Does one million tokens mean one million tasks?

No. A task can use hundreds to thousands of tokens. The cost of a million tasks is the per-token price multiplied by measured or explicitly assumed usage.

Does Gemini 3.8 Flash lose in the Decision Index?

There is no basis for that conclusion. As of October 13, 2026, Gemini 3.8 Flash had no score in the index. No measurement is not the same as a loss.

Are GPT-6 Luna and GPT-6 Luna Decisions the same result?

No. AA lists the language model separately (low: 55.2 points; non-reasoning: 49.6) from the Decisions variant (48.5 points). Each has its own cost and latency.

Is an index score the percentage of decisions made correctly?

No. It combines four different metrics. For a specific use case, choose the relevant component and test the model on representative data.

Disclosure: I know Jev from earlier research. In September, I wrote a simple API client for it for use with my own editorial tools. TypeSafe did not commission or pay for this article. My assessment is based on independent Artificial Analysis measurements and my own test, in which every model received the same tasks.

My tests: October 13, 2026 (50 Polish support tickets, 11 models through OpenRouter from Poland) and October 5, 2026 (30 Polish support tickets, including Gemini 3.8 Flash through the API).

Kinetyra