Can machine learning find mislabeled data? We planted 711 wrong labels to measure it
We planted 711 wrong labels in 14,214 products. Cross-checking flagged 710; 95.4% of flags at confidence 0.9+ were planted errors. See the method.
Read the report →STAYCHARTED BENCHMARKS
Compare measured results from StayCharted AI Model Trainer. Choose a topic to find the relevant scores, methods, and full reports.
16 reports
Compare the model options in StayCharted AI Model Trainer. Scores are measured accuracy before human review.
| Dataset / task | StayChartedAI Model Trainer | External comparison | Report | ||
|---|---|---|---|---|---|
| ClassifierWords | AI ClassifierPre-trained AI | Dedicated AIFine-tuned AI | ClaudeSonnet 5.5 | ||
| Amazon Access SamplesGroup access · 10 yes/no decisions | 97.8% | — | — | — | View → |
| Wayfair WANDS product categories158 categories · 2,844 products | — | 95.3% | 97.3% | — | View → |
| BANKING77Support messages · 77 categories | — | 92.6% | 93.7% | 84.7% | View → |
| NHTSA vehicle systems21 categories · 5,000 later complaints | — | 84.1% | 87.7% | — | View → |
| NHTSA subsystems70 test categories · 5,000 later complaints | — | 76.9% | 85.9% | — | View → |
| LEDGARContract clauses · 100 categories | — | 82.7% | 84.9% | 76.9% | View → |
| WANDSSearch relevance · 3 labels · 4,000 pairs | — | 55.6% | 80.2% | 72.7% | View → |
| Medical AbstractsResearch abstracts · 5 categories | — | 60.0% | 63.6% | 65.7% | View → |
| Amazon ESCISearch relevance · 4 labels · 4,000 pairs | — | 44.2% | 59.0% | 59.9% | View → |
| CBP HS codes · 6-digit subheading1,167 later rulings | — | 29.9% | 39.2% | — | View → |
| CBP HS codes · exact 10 digitsOne-line description · 2,014 codes · 1,167 later rulings | — | 22.2% | 25.1% | — | View → |
Claude received category definitions; StayCharted models learned from labeled examples. Medical Abstracts has overlapping categories, and Claude scored highest there. Compare the methods → · Search relevance method and query split →
| Dataset / task | StayChartedAI Model Trainer · Image models | Report | |
|---|---|---|---|
| AI Image ClassifierPre-trained AI | Dedicated AI ImageFine-tuned AI | ||
| FruitNetFruit quality · 2 grades per fruit | 98.7% | 98.7% | View → |
| Cat and dog breedsBreed categories · 37 | 95.2% | — | View → |
| VisA cashewsDefect detection · 2 categories | 90.0% | 93.8% | View → |
| Catalog product photosProduct categories · 40 | 91.1% | — | View → |
| Car exterior parts21 parts · 840 crops | 85.5% | 89.9% | View → |
| Part numbers from a photo100 look-alike parts · 1,000 photos | 87.8% | 75.3% | View → |
| Car damageSix categories · 552 photos | 76.6% | 86.4% | View → |
| Car damage types8 types · 449 crops | 64.1% | 70.2% | View → |
On 100 look-alike parts with 20 training photos each, the AI Image Classifier Model beat the Dedicated AI Image Model. Limited examples per part may explain the fine-tuned model’s lower score in this setup. See the method and dataset attribution →
For cashews, the AI Image Classifier found 16 of 40 defects (40%); the Dedicated AI Image Model found 26 (65%). Overall accuracy also includes sound items.
Each table is sorted by the best score in its row. Every benchmark we've run is listed, including the hard ones.
Bold marks the highest accuracy among the compared results in each row, including ties. — means not reported in this summary. Full reports contain test sizes, dates, error breakdowns and review projections; results across different datasets are not directly comparable.
We planted 711 wrong labels in 14,214 products. Cross-checking flagged 710; 95.4% of flags at confidence 0.9+ were planted errors. See the method.
Read the report →StayCharted classified 5,000 later NHTSA complaints: 87.7% system and 85.9% subsystem accuracy. Compare models, confidence, mistakes and the test method.
Read the report →StayCharted identified 87.8% of 1,000 unseen photos across 100 part numbers; 99.7% accuracy on the 32.3% above confidence 0.5. Compare both image models.
Read the report →StayCharted AI Classifier Model: 95.3%; Dedicated AI Model: 97.3% on the same 2,844 products across 158 categories. Dedicated default cutoff: 95.5% coverage at 99.0% accuracy.
Read the report →We trained two models on 5,621 CBP rulings and tested them on 1,167 later ones. Accuracy at 10, 6, 4 and 2 digits, and how often the surest answers are right.
Read the report →Compare StayCharted’s image models on car damage, 21 exterior parts and eight damage types. See measured accuracy, rare-category errors and review projections.
Read the report →Compare StayCharted and Claude on WANDS and ESCI: 4,000 query-product pairs per test, accuracy by category, review projections and train-test leakage.
Read the report →StayCharted trained on 26,103 access decisions and tested on 6,666 held-out requests. Compare routine workload, review routing and missed denials.
Read the report →Compare trained models for access permissions. See held-out accuracy, review thresholds and results for learning group membership from past decisions.
Read the report →Compare Claude Sonnet 5.5 with StayCharted trained classifiers on banking messages, contract clauses and medical abstracts. See all results, method and limits.
Read the report →Medical abstract classification for life sciences literature triage: word models, embeddings and a fine-tuned model compared on published abstracts.
Read the report →Food grading by photo with StayCharted AMT: compare fruit quality accuracy, cashew defect detection, and projected human review on FruitNet and VisA.
Read the report →Compare contract clause classification on LEDGAR: words, embeddings and fine-tuning across accuracy, training time and projected human review.
Read the report →SigLIP 2 scored 91.1% on catalog photos versus DINOv2’s 88.3–88.8%; DINOv2 base led on pet breeds. Compare StayCharted’s image-classifier tests.
Read the report →See how AI Classifier Models compare meaning alone with words plus meaning and keep the better validation result. Explore measured results and tradeoffs.
Read the report →Banking Intent Classification with StayCharted AMT: 93.7% accuracy on the BANKING77 test set, plus confidence thresholds and human-review tradeoffs.
Read the report →A score describes a model on a particular dataset and test split. Check the categories, examples, model type and errors before applying the result to your task. Accuracy before review and projected accuracy after review are different measurements.
STAYCHARTED AI MODEL TRAINER
Start with examples you already have, see how often it’s right on items it never saw, and use confidence to prioritize the ones a person should review.
Start free