STAYCHARTED BLOG · Machine learning · Data quality
Machine learning for data cleansing: finding the wrong categories in your own records
Machine learning for data cleansing: a model trained on your own records flagged 710 of 711 wrong labels we planted in 14,214 products. How it works.
Which data cleansing jobs machine learning is good at
Data cleansing covers many jobs. Machine learning is the right tool for some of them and the wrong one for others.
| Job | This classification workflow? | Why |
|---|---|---|
| Find records filed under the wrong category | Yes | It needs to know what each category looks like, which is what a trained model learns |
| Put free text onto a standard list (supplier descriptions to your taxonomy, notes to failure codes) | Yes | Same skill: reading text and choosing one of your categories |
| Fill a blank category column | Yes | A fill writes the model's category, with a confidence, on every row |
| Spot category names that mean the same thing ("Billing" and "billing issue") | Partly | A data quality check does this before training |
| Standardize formats (dates, phone numbers, addresses) | No | Rules and formatting tools do this exactly |
| Match the same customer or vendor across systems | No | That is record matching, a different technique |
| Fill missing numbers, find numeric outliers | No | Statistical tools and your warehouse handle these |
The rest of this post is about the first three, where training on your own decisions pays off.
How data cleaning with machine learning finds wrong labels
The idea is simple. A model learns what each category usually contains. When it reads a record and confidently answers a different category from the one on file, that record is worth a look. Two details make the difference between a useful list and a useless one.
Never check rows with the model that learned them
A model asked about its own training rows mostly repeats their labels, wrong ones included. In our test, the same model checking the rows it was trained on caught only 27.1% of the planted errors. Split your data in two, train on one half, fill the other, then swap. Every row gets judged by a model that never saw its label. Done that way, 710 of the 711 errors were flagged.
Review the surest disagreements first
About half of all disagreements in our test were planted errors. The rest were the model's mistakes or real errors already in the data. Sorting by confidence puts the real errors at the top:
| Flag when the model's confidence is | Rows flagged | Wrong labels caught | Flags that were wrong labels |
|---|---|---|---|
| any | 1,463 | 710 of 711 | 48.5% |
| 0.7 or higher | 419 | 378 | 90.2% |
| 0.9 or higher | 151 | 144 | 95.4% |
For a fixed amount of review time: checking the surest 500 rows (3.5% of the data) found 436 wrong labels. A random sample of 500 rows would hold about 25. And when the model flagged a wrong label, its own answer was the original category 94.9% of the time, so the reviewer usually just confirms the fix.
Expect to find errors you didn't know about
The model also flagged rows we hadn't touched. Among the 50 it was surest about, our reading is that about 9 were real errors in Wayfair's own data, such as a kitchen faucet filed under Kitchen Sinks. About 23 sat between categories that overlap, such as wastebaskets under both Bath Accessories and Trash Cans & Recycling. That second group is a finding too: when two categories both fit, a one-line rule in your labeling guide fixes more rows than correcting them one at a time.
AI data cleansing for your industry
Potential uses include the following. The benchmark measured product labels, not these other tasks:
| Industry | Records worth checking |
|---|---|
| Retail and distribution | Product categories, especially after supplier feeds and catalog merges (product categorization) |
| Automotive and manufacturing | Warranty claims and repair orders coded to the wrong failure or part (warranty claim classification) |
| Finance and procurement | Spend categories and general ledger coding (spend classification) |
| Customer support | Ticket categories that drifted as teams and queues changed |
| Trade and logistics | HS codes and product master data (HS code classification) |
| Insurance | Claim types and loss causes coded differently by different adjusters |
Cleaning AI training data
Wrong labels matter even more when the data trains another model, because the model learns the mistakes and repeats them at scale. The same cross-check works before any fine-tuning or classifier training: train on half, fill the other half, swap, and review the surest disagreements. In AMT, you can then correct the labels and retrain, and the Review Queue on Essentials and Business sends uncertain new answers to a person. The high-confidence disagreement list in this cross-check is prepared separately from that queue. For data that has no labels yet, see pre-labeling training data.
Product data cleansing in AMT, step by step
- Export your records with the text columns and the category column, as CSV or Excel, or connect a Snowflake or Databricks table.
- Split them in two at random, half A and half B.
- Train an AI Classifier Model on each half. Free includes one AI Classifier Model. Use Essentials or Business for two models at once and sufficient capacity for this full benchmark. Before training, AMT's data quality check shows category names that look like the same thing, near-duplicate rows and personal information, and you choose what to merge, remove or mask.
- Fill the other half with each model: half A's model fills half B with the category column blank, and the other way round. Each row gets a category and a confidence.
- List the disagreements, highest confidence first, and review from the top. Accept the model's answer, keep the label, or note an overlap for your labeling guide.
- Fix the labels and retrain one model on the cleaned data. That's the model to use on new records.
AMT doesn't run steps 2 to 5 for you in one click today; it's two models and two fills, as in our benchmark. Start free, see plans, or read how AMT works.
When you don't need machine learning
If you have a few hundred records, read them. If your categories are few and obvious, a filter or a pivot table finds the odd ones faster. And if the problem is formats or duplicates, use the tools made for those. Machine learning earns its place when there are thousands of records, many categories and years of different people filing them.
Common questions
Can machine learning be used for data cleansing?
Yes, for the parts that need judgment about categories: finding records filed under the wrong category, mapping free text to a standard list and filling blank categories. For formats, duplicates across systems and missing numbers, rules and data preparation tools are a better fit.
How does machine learning find mislabeled data?
A model trained on part of your records reads the rest and answers a category for each. Where its confident answer differs from the label, the label is likely wrong. In our benchmark this flagged 710 of 711 planted errors, and 95.4% of the flags at 0.9 confidence or higher were planted errors.
How many records do I need to review?
Far fewer than all of them. In our test, reviewing the surest 500 flags (3.5% of the data) found 61% of the wrong labels, and the surest 1,000 (7%) found 91%.
Why not check the data with the model trained on it?
It has learned the wrong labels and mostly repeats them. Checking its own training rows caught only 27.1% of the errors in our test. Train on one half and check the other.
Is AI data cleansing safe for sensitive data?
AMT checks training data for personal information such as emails, phone numbers and card numbers, and lets you mask or remove it before training. Detection does not catch every sensitive detail; review and remove anything it misses. Your workspace’s categories are learned from your examples. See security.
Does it fix the errors automatically?
No. It suggests a fix for each flagged row, and its suggestion was the original category 94.9% of the time in our test. A person decides, because some flags are the model's mistakes and some are categories that overlap.