AI confidence score accuracy: how often sure answers were right in our benchmarks

A model is well calibrated when its confidence tracks how often it is right. For a confidence threshold (the cutoff above which answers go straight through), two numbers matter:

• Coverage: the share of answers at or above the cutoff. This is the share eligible to bypass review if your workflow permits it.
• Right among those: the share of answers above the cutoff that were correct. The error rate among those answers is 100% minus this figure.

Raising the cutoff usually lowers coverage and raises the share right. Usually is the key word. Every model StayCharted AI Model Trainer (AMT) trains is tested on held-out rows it never saw, and the report shows the share of answers above each cutoff with how many of those were right. That table is where you find out whether your model behaves.

Accuracy above a cutoff is useful for choosing review thresholds, but it is not a full calibration test. A well-calibrated 0.9 score would correspond to about 90% correct across many answers near that score; the tables here combine all answers above the cutoff.

Here is the 0.9 cutoff across the benchmarks that report it. Each figure is from the linked benchmark page, measured on held-out test rows.

Most rows show a confidence you can work with. Two do not: the Dedicated AI Model on HS codes and the warehouse function on 72 subsystems.

The CBP benchmark tested 1,167 later rulings on exact 10-digit codes from a one-line subject. The Dedicated AI Model was more accurate overall (25.1% against 22.2% at 10 digits), yet its confidence was the less useful one.

At 0.9 the AI Classifier Model answered 78 rulings and got three wrong. The Dedicated AI Model's top band was 15 rulings, and its answers at 0.9 were less often right than its answers at 0.7. A team that copied a 0.9 cutoff from another task onto that model would have let through answers that were wrong more often than right. These subsets are small and from a single run, so each error moves the percentage. These figures should not be treated as a formal significance test.

The benchmark page does not say why. The HS test has 2,014 codes in training, more than half of them seen only once, which is the kind of data where we would check calibration before trusting any cutoff.

On WANDS product categories the pattern reversed. The Dedicated AI Model sent 95.5% of products straight through at its default 0.9 cutoff, and 99.0% of those were right. The AI Classifier Model's 0.9 answers were right 99.7% of the time but covered only 41.4%; at its default 0.5 cutoff it covered 85.2% with 98.8% right.

On NHTSA complaints, both models' answers at 0.90 were right 93.7% to 96.7% of the time, on complaints received from 2025 onward. The Dedicated AI Model covered far more of the work (84.7% of vehicle-system answers against 54.7%), while the AI Classifier Model was slightly more often right among the answers it was sure about.

The part-identification benchmark used 100 look-alike parts. The AI Image Classifier Model named the exact part on 87.8% of test photos, and the Dedicated AI Image Model on 75.3%. The page puts it plainly: "With 100 look-alike parts, the AI Image Classifier Model is rarely very sure, even when it's right."

A 0.3 from the AI Image Classifier Model was more trustworthy here than a 0.9 from the Dedicated AI Image Model. The page concludes that for the same amount of checking, the AI Image Classifier Model was better at every cutoff. A confidence number does not carry over from one model to another.

Our Snowflake and Databricks comparison used the same NHTSA complaints. Snowflake AI_CLASSIFY returns labels with no confidence, so no cutoff can be set. Databricks ai_classify does return one:

• On 21 vehicle systems at 0.80, it covered 28.6% of complaints with 94.9% right (names alone), or 38.7% with 95.1% right (with examples).
• On 72 subsystems at 0.80, it covered 33.2% with 56.4% right.
• On 72 subsystems at 0.90, it covered about 2.8% with 62.3% right.

On the same 72 subsystems, the Dedicated AI Model covered about 71.5% at 0.90 with 94.3% right. On a few broad categories the warehouse function's confidence was usable; on many close ones it was not.

Across these tables:

• AI Classifier Model: in all four text tables above, its answers at 0.9 or higher were right 94.8% of the time or more. Coverage ranged from 6.7% (HS codes) to 54.7% (vehicle systems), so it is careful rather than generous.
• Dedicated AI Model: usually covers much more at 0.9 (95.5%, 84.7% and 71.5%) with 93.7% to 99.0% right. On HS codes it did not hold up, at 46.7%.
• AI Image Classifier Model: on part photos, low confidence numbers but very reliable ones.
• Dedicated AI Image Model: 97.9% right at 0.9 on part photos and 98.4% on damaged or not.

No model kind was best everywhere, which is why the table for your data matters more than any general rule.

1. Train your model in AMT and open its test report. Find the table of cutoffs, share above each, and share right.
2. Decide the error rate you can accept on answers that skip review. That depends on the cost of a wrong answer, not on the model.
3. Pick the lowest cutoff whose share right meets that bar, and read its coverage. That is how much work goes straight through.
4. Check that the share right rises as the cutoff rises. If the top band is less often right than a lower one, as with the Dedicated AI Model on HS codes, do not rely on that band.
5. Never copy a cutoff from another model kind or another task. Defaults differ too: 0.5 for the AI Classifier Model and 0.9 for the Dedicated AI Model in these benchmarks.
6. Send answers below the cutoff to the Review Queue (Essentials and Business plans). Reviewers' answers train the next version, and the new version gets a new table, so read it again after each retrain. See human-in-the-loop machine learning.

Trust them when the held-out table for your own data shows a share right you can live with, at a coverage worth having, and when a wrong answer is cheap to fix later. Some work, such as a declared customs code, still needs a person who owns the result whatever the confidence says. For every benchmark see all benchmarks, for plans see pricing, and for features see the product page.