AI Confidence Scores: What 0.9 Means and How to Check Calibration

👁 117 reads

A confidence score of 0.9 sounds like a promise: the model should be right nine times out of ten. But the number alone proves nothing. You need to compare it with correct answers on data like the cases your application will handle. Below, I explain calibration, show calculations from saved results for three models, and share a simple Python script. The key point: a confidence threshold and an acceptable error rate are two different things.

What an AI confidence score of 0.9 really means

Calibration is a property of many predictions. If you collect answers with confidence close to 0.9, a well-calibrated model should get about 90% of them right. That does not mean an individual answer has been verified. A model can also be well calibrated while having low accuracy: it can honestly signal that it is often wrong.

Guo and co-authors describe this distinction in their 2017 paper on neural network calibration. Improving accuracy alone is not enough to make probabilities reliable. For an application, both choosing the right answer and using confidence to identify cases that need review matter.

Why chatbots usually do not show a confidence score

In decision models, answers come from a closed set. A model can return a probability distribution over the options, and the confidence in its choice is usually the probability assigned to the selected answer. That gives you a specific result to check against a known answer key.

Language models also operate on probabilities. However, logprobs, available in some interfaces, describe token probabilities, not whether an entire sentence is true. Structured output sets the response format, such as JSON containing a category name. The format alone does not provide calibrated confidence. A confidence field added by a chatbot also needs to be verified. So it is not true that chatbots cannot provide confidence scores; the challenge is what the score means and how to measure it.

Calibration plot based on actual results

On 19 October 2026, I calculated the agreement between kat and gold_kat, along with the average conf, in three output files. Basal-omni-1.0-mini Q8_0 was tested locally on 19 October; Jev 1.13 and Perplexity Decider v1.1 27B were tested via API on 13 October. Each model has all 50 cases, with no difficult cases removed; I exclude separate tool-selection tasks. This is an evaluation against the saved answer key, including cases where a submission may be ambiguous.

The lower bound of each range is included and the upper bound excluded, except for the final range, which also includes 1. Empty ranges are omitted. Overall accuracy is 42/50, 41/50 and 44/50, respectively. Do not confuse these denominators with the results for 47 cases described in the local basal test.

In a calibration plot, the horizontal axis shows a group’s mean confidence and the vertical axis its accuracy. Ideal points lie on the diagonal. A point below it indicates overconfidence; a point above it indicates underconfidence. For basal’s highest range, the point has coordinates of 97.7% and 87.5%: it is clearly separated from the diagonal.

ECE: calibration error weighted by the number of answers

ECE is the weighted average of the absolute differences between accuracy and confidence across groups, with each group’s weight determined by its size. The basal group with 40 answers has a weight of 40/50. Its difference is about 10.17 percentage points, contributing about 8.14 points to ECE. Including the other groups gives a result of 0.1302, or 13.02 percentage points.

Using the same range boundaries, Jev has an ECE of 0.1224 and Decider 0.0861. A lower score means a smaller measured discrepancy, but it does not prove better performance on other data. ECE depends on how you divide results into groups, and individual observations are unstable. It is also not the error rate or a Calibration score from another benchmark.

Check automation thresholds separately

First set the maximum acceptable error rate among decisions handled automatically, such as 5% or 1%. Then choose a threshold using tuning data and lock it before evaluating it on an independent dataset. Measure two things: the error rate among accepted cases and coverage, meaning the share of all cases handled automatically.

At a threshold of 0.9, basal accepts 40/50 cases but gets 5/40 wrong, or 12.5%. Decider accepts 41/50 and gets 2/41 wrong, about 4.88%. That second result meets the 5% target only as an observation from a small sample. It is not statistical confirmation of the target risk.

• Set aside whole groups of related cases so variations of the same example do not appear in both tuning and test data.
• Check important categories separately. A good overall result can hide poor performance on a rare class.
• Repeat the measurement after changing the model, instructions, quantization or data type.

How many examples do you need?

Fifty examples can reveal a problem, but cannot confirm a 1% error rate. With a threshold set in advance, independent observations and zero mistakes, the one-sided 95% confidence upper bound on the error rate is 1 - 0.05**(1/n). You need at least 59 accepted decisions for a 5% bound and 299 for a 1% bound. These are counts of accepted decisions, not all the examples collected; errors increase the sample size needed.

Basal’s documentation also separates fitting from evaluation and accounts for statistical uncertainty. See the basal-calibrate guide for instructions on using that specific tool. The principle applies across models: do not choose a threshold using the same answers you later use to justify its performance.

Python: calculate calibration from a file

Save the code below as calculate_calibration.py. It uses only the standard library and the format used in the output files described here. Run python calculate_calibration.py results.json 0.9 to display the groups, ECE, coverage and observed error rate. The script does not calibrate a model or certify a threshold.

import json
import math
import sys

with open(sys.argv[1], encoding="utf-8") as f:
    rows = json.load(f)["wyniki"]
rows = [r for r in rows if r.get("typ") == "zgl"]
if not rows:
    raise ValueError("No cases found")
for r in rows:
    c = r["conf"]
    if not math.isfinite(c) or not 0 <= c <= 1:
        raise ValueError("Confidence out of range")
    r["ok"] = r["kat"] == r["gold_kat"]

edges = [0, 0.5, 0.7, 0.9, 1]
ece = 0
for lo, hi in zip(edges, edges[1:]):
    group = [r for r in rows
             if lo <= r["conf"] < hi
             or hi == 1 and r["conf"] == 1]
    n = len(group)
    if not n:
        continue
    conf = sum(r["conf"] for r in group) / n
    acc = sum(r["ok"] for r in group) / n
    ece += n / len(rows) * abs(conf - acc)
    print(lo, hi, n, round(conf, 4), round(acc, 4))
print("ECE:", round(ece, 4))

threshold = float(sys.argv[2])
if not 0 <= threshold <= 1:
    raise ValueError("Threshold out of range")
accepted = [r for r in rows
            if r["conf"] >= threshold]
print("Coverage:", len(accepted) / len(rows))
if accepted:
    errors = sum(not r["ok"] for r in accepted)
    print("Error rate:", errors / len(accepted))
else:
    print("Error rate: no accepted decisions")

Start by checking the group sizes. A 100% result from two answers tells you very little. Only further independent examples will show whether the number returned by the model is actually suitable for making decisions.

Calculation source: local files wyniki_basal_mini_q8.json, wyniki_jev113.json and wyniki_decider11.json; recalculated on 19 October 2026.

// read moreAI Ticket Routing in Python with Strands Decider 2BGuidesLLM Confidence Scales: 0-20 Works Better Than 0-100BenchmarksIBM Research Tool Calibrates AI Agent Memory RequirementsNews

Kinetyra