STAYCHARTED BENCHMARKS

AI classification benchmarks on real data

Compare measured results from StayCharted AI Model Trainer. Choose a topic to find the relevant scores, methods, and full reports.

16 reports

Measured results at a glance

Compare the model options in StayCharted AI Model Trainer. Scores are measured accuracy before human review.

Text and access classification

Dataset / taskStayChartedAI Model TrainerExternal comparisonReport
ClassifierWordsAI ClassifierPre-trained AIDedicated AIFine-tuned AIClaudeSonnet 5.5
Amazon Access SamplesGroup access · 10 yes/no decisions97.8%———View →
Wayfair WANDS product categories158 categories · 2,844 products—95.3%97.3%—View →
BANKING77Support messages · 77 categories—92.6%93.7%84.7%View →
NHTSA vehicle systems21 categories · 5,000 later complaints—84.1%87.7%—View →
NHTSA subsystems70 test categories · 5,000 later complaints—76.9%85.9%—View →
LEDGARContract clauses · 100 categories—82.7%84.9%76.9%View →
WANDSSearch relevance · 3 labels · 4,000 pairs—55.6%80.2%72.7%View →
Medical AbstractsResearch abstracts · 5 categories—60.0%63.6%65.7%View →
Amazon ESCISearch relevance · 4 labels · 4,000 pairs—44.2%59.0%59.9%View →
CBP HS codes · 6-digit subheading1,167 later rulings—29.9%39.2%—View →
CBP HS codes · exact 10 digitsOne-line description · 2,014 codes · 1,167 later rulings—22.2%25.1%—View →

Claude received category definitions; StayCharted models learned from labeled examples. Medical Abstracts has overlapping categories, and Claude scored highest there. Compare the methods → · Search relevance method and query split →

Image classification

Dataset / taskStayChartedAI Model Trainer · Image modelsReport
AI Image ClassifierPre-trained AIDedicated AI ImageFine-tuned AI
FruitNetFruit quality · 2 grades per fruit98.7%98.7%View →
Cat and dog breedsBreed categories · 3795.2%—View →
VisA cashewsDefect detection · 2 categories90.0%93.8%View →
Catalog product photosProduct categories · 4091.1%—View →
Car exterior parts21 parts · 840 crops85.5%89.9%View →
Part numbers from a photo100 look-alike parts · 1,000 photos87.8%75.3%View →
Car damageSix categories · 552 photos76.6%86.4%View →
Car damage types8 types · 449 crops64.1%70.2%View →

On 100 look-alike parts with 20 training photos each, the AI Image Classifier Model beat the Dedicated AI Image Model. Limited examples per part may explain the fine-tuned model’s lower score in this setup. See the method and dataset attribution →

For cashews, the AI Image Classifier found 16 of 40 defects (40%); the Dedicated AI Image Model found 26 (65%). Overall accuracy also includes sound items.

Each table is sorted by the best score in its row. Every benchmark we've run is listed, including the hard ones.

Bold marks the highest accuracy among the compared results in each row, including ties. — means not reported in this summary. Full reports contain test sizes, dates, error breakdowns and review projections; results across different datasets are not directly comparable.

Explore the reports

Read each result in context

A score describes a model on a particular dataset and test split. Check the categories, examples, model type and errors before applying the result to your task. Accuracy before review and projected accuracy after review are different measurements.

STAYCHARTED AI MODEL TRAINER

Train AI to categorize the way your team does.

Start with examples you already have, see how often it’s right on items it never saw, and use confidence to prioritize the ones a person should review.

Start free