Can machine learning find mislabeled data? We planted 711 wrong labels to measure it

Every team that categorizes things by hand has some wrong labels. They teach a model the wrong thing, skew reports, and are hard to find, because no one can reread 14,000 rows. We took 14,214 Wayfair products in 158 categories, changed 711 of their categories (5%) to wrong ones on purpose, and asked whether a model trained on that same flawed data could point to them.

• It found almost all of them. Flagging every row where the model disagreed with the label caught 710 of the 711 planted errors. That list was 1,463 rows (10.3% of the data), and about half of it was the planted errors.
• Confidence puts the real errors first. Of the 151 flags where the model was at 0.9 confidence or higher, 95.4% were planted errors, and for those 144 planted errors, the model's answer was the original category every time.
• A small review finds most of them. Reviewing the 500 surest flags (3.5% of the data) found 436 planted errors. A random sample of 500 rows would hold about 25.
• How you check matters more than which model. The same model checking rows it had trained on caught only 27.1%: it had learned the wrong labels. Each half has to be checked by a model trained on the other half.
• It also flagged possible errors in the original data. Among flags on rows we did not touch, several look like labels Wayfair's data already had wrong, such as a "1-light flush mount" filed under Wall Sconces.

Why two halves. A model asked about rows it was trained on mostly repeats their labels, wrong ones included. Training on one half and checking the other means every row is judged by a model that never saw its label. This is the standard idea behind label-error detection in machine learning research (cross-validation); here it is done with two ordinary models and two fills.

Why plant errors. Real data has wrong labels too, but no one knows exactly which. Planting a known 5% lets us count exactly how many were found.

• StayCharted AI Classifier Model, crossed halves: a ready-made AI reads the meaning of each product, and a classifier learns the 158 categories. Trained and filled on StayCharted's production service through the same screens a customer uses. On AMT's own held-out test (whose labels are also 5% wrong) the two halves scored 89.8% and 90.0%.
• A words-only model, crossed halves: TF-IDF word and character features with logistic regression, run outside AMT, as a reference.
• StayCharted AI Classifier Model, same rows: each half filled by the model trained on that same half. This is the mistake to avoid, measured so you can see why.

StayCharted AI Classifier Model, crossed halves

Both kinds of error were caught: 99.7% of the same-department slips and 100% of the careless ones. The model's suggested fix was the original category on 93.7% and 96.1% of them.

The other half of the flags. About half of the 1,463 flags were not planted errors. Some are the model's mistakes, and some may be errors already in the data, discussed below. That is why the review list should be read surest first, and why a person decides.

The final column measures corrections among the planted errors caught, not among every flag. The unchanged rows may include uncertain or overlapping original labels.

Sort the flags by the model's confidence and review from the top:

Compared with the words-only model and with the same-rows mistake

• The two models are close. Product titles are full of telling words ("faucet", "rug"), so a words-only model does nearly as well here. The AI model's surest flags are slightly more precise; the words-only model is sure of more of them. On text where meaning matters more than keywords, such as support messages or claims, we would expect the gap to be wider, but we did not measure that here.
• The same-rows mistake is costly. It finds a quarter of the errors, and the ones it does find are probably mostly from the 15% of rows AMT held out of training for its own test.

The model also flagged 753 of the 13,503 rows we did not change (5.6%). We read the 50 it was surest about, by product name only. This is our judgment, not a measurement:

• About 9 look like real errors in the original data.
  • "kelly 1-light flush mount", filed as Wall Sconces (the model said Flush Mount Lighting, 0.98)
  • a "4-tier multipurpose wide shelving unit", filed as TV Stands & Entertainment Centers
  • a kitchen faucet, filed as Kitchen Sinks
  • a tissue box cover, filed as Toilet Paper Holders
  • a garden statue, filed as Outdoor Wall Decor
• About 18 are the model's mistakes. For example, Kohler shower trim sold "valve not included" was answered as Fixture Parts And Components, and a mattress topper as Foam and Latex Mattresses.
• About 23 sit between categories that overlap.
  • Wastebaskets appear under both Bath Accessories and Trash Cans & Recycling.
  • Kids' daybeds appear under both Kids Beds and Daybeds & Guest Beds.
  • Welcome signs appear under both Holiday Accents & Decor and Outdoor Wall Decor.

Those overlaps are cleansing work too. When two categories both fit, people file the same product both ways, and the model learns the inconsistency. A one-line rule for each overlap, kept in a labeling guide, fixes more rows than correcting them one by one.

• Check with crossed halves. Train one model on half your rows and fill the other half, then swap. Never check rows with the model that trained on them.
• Review surest first, and decide how deep to go. The top few hundred flags are mostly real errors; further down, more of them are the model's mistakes.
• Use the model's answer as the suggested fix. When it flagged a planted error, its answer was the original category 94.9% of the time.
• Fix, retrain, and evaluate the next model. Corrected labels can improve suggestions on new rows; compare the new version on reviewed examples before publishing it.
• Write down the overlaps. Most of the disagreements that weren't errors came from categories that overlap.

In AMT today, the cross-check is two models and two fills, as in this test. AMT's data quality check before training finds category names that look like the same thing, near-duplicate rows and personal information. For text it does not yet look for wrong labels: it found none of the 711.

• Data: WANDS products with a single product class and at least 50 products per class, at most 100 per class, as prepared for our WANDS product categorization benchmark. Train and test sets from that benchmark were combined.
• Planting: products were chosen by a fixed hash. Even-numbered picks moved to a random other category in the same top-level department when one existed; the rest moved to a random other category.
• Training and filling:
  • Two StayCharted AI Classifier Models, trained on StayCharted's production service on 1 November 2026, about 25 minutes each.
  • Published and filled through the app; each fill of about 7,000 rows took about 23 minutes.
  • The fill files had the category column blank.
• Scoring: a row is flagged when the filled category differs from the given (possibly planted) one. "Caught" counts flagged planted rows. "The original category" compares the model's answer with the label before planting.
• Words-only reference: scikit-learn TF-IDF (word 1 to 2-grams and character 3 to 5-grams) with logistic regression, trained on each half and applied to the other.
• One run: each figure comes from one split and one set of planted errors. Results may differ with another split, dataset or pattern of errors.
• Not measured: other datasets, other error rates, or the Dedicated AI Model.

Yan Chen et al. WANDS: Dataset for Product Search Relevance Assessment, ECIR 2022. Dataset and MIT license.

Use the cross-check on your own records.

Can machine learning find mislabeled data?

Yes, if the model checking a row didn't learn that row's label. In this test, models trained on one half of 14,214 products flagged 710 of the 711 wrong labels we planted in the other half. 48.5% of all flags were planted errors, rising to 95.4% at confidence 0.9 or higher.

How do you detect label errors in a dataset?

Train a model on part of the data, use it on the rest, and look at rows where its answer disagrees with the label, starting with the ones it's most confident about. Swap the parts so every row gets checked. A model checking its own training rows misses most errors: 27.1% caught in our test.

What share of flagged rows are real errors?

In this test, 48.5% of all disagreements were planted errors, 81.5% of those at 0.5 confidence or higher, and 95.4% at 0.9 or higher. Some of the rest were real errors in the original data.

How many rows do I need to review?

Far fewer than the whole dataset. Reviewing the surest 500 flags (3.5% of the data) found 61% of the planted errors, and the surest 1,000 (7%) found 91%.

Does it need an AI model, or will a simple one do?

On product titles, a simple words-only model did nearly as well (75.5% vs 77.9% of the errors in the same review budget). Product names are full of telling words. Where wording varies more, a model that reads meaning should help more, but this test didn't measure that.

Is the data public?

Yes. WANDS is published by Wayfair under the MIT license.