AI model drift: what happened when we trained on the past and tested on the future

The usual way to test a classifier is to hold back a random share of the labeled rows. That is fine for checking that a model learned something, but it mixes old and new records on both sides. The HS code benchmark page explains why it avoided this: a random split would put rulings from the same week on both sides, and "The test would then reward remembering."

A time-based split trains on everything up to a date and tests on what came after. It is the closest thing to real use, because every row your model fills tomorrow is newer than its training data. The NHTSA benchmark page calls its date split "a harder and more honest test than a random split."

The vehicle-complaint benchmark trained two models at two levels of detail:

• Vehicle system (21 categories): 19,902 complaints received 2023 to 2024.
• Subsystem (72 categories in training, 70 of them in the test): 19,281 complaints received 2020 to 2024.

Both levels were tested on 5,000 complaints received from 1 January 2025 onward. The page also reports scores on complaints held back from the training years, which gives a before-and-after view:

The observed score changes were modest. This comparison does not isolate drift from differences in the test samples. The biggest drop was the AI Classifier Model on the finer subsystem level. The Dedicated AI Model's subsystem score went the other way, and the page does not explain why, so we do not claim a reason. The page's own reading of why newer complaints are harder: "New vehicles and features bring new ways of describing problems, which is why the unsure answers go to a person and their decisions train the next version."

Confidence held up on the newer complaints

The confidence tables on that page were measured on the 2025-onward complaints, so they already include the drift. At a 0.90 cutoff:

Even on records from a later period, the answers the models were sure about were right at least 93.7% of the time. That is the case for a confidence cutoff with a review step: drift lowered overall accuracy a little, while the answers above the cutoff stayed reliable on the later records and the unsure ones went to a person. For how to read these tables, see AI confidence score accuracy.

The customs benchmark trained on 5,621 CBP rulings from 29 March 2021 to 18 December 2025 and tested on 1,167 later rulings, from 18 December 2025 to 14 August 2026. Here drift showed up in a different form: new categories.

• 27% of test rulings have a code that never appears in training. As the page says, a model "can't suggest a code it has never seen."
• Across all 1,167 test rulings, the AI Classifier Model matched the exact 10-digit code 22.2% of the time and the Dedicated AI Model 25.1%.
• On the 849 rulings whose code does appear in training, the 10-digit figures are 30.5% for the AI Classifier Model and 34.5% for the Dedicated AI Model.

That gap is the cost of new categories. The models did better on codes they had seen, and every ruling with an unseen code counted as wrong. The NHTSA test had no such problem at the subsystem level: all 70 test subsystems were among the 72 in training.

It can, and how much depends on what changes. These two tests show two kinds of change:

• New wording for the same categories. NHTSA complaints from 2025 onward used the same subsystems, described in newer ways. Accuracy fell by up to 3.4 points on three scores and rose on one.
• New categories. In the CBP test, more than a quarter of later rulings needed a code the training data did not contain. These fixed-category classifiers need examples of the new labels before they can predict them.

A fixed model can fall behind when these changes affect its task. The first kind tends to be gradual. The second is sudden: every row that needs the new category gets a wrong answer, because the right category is not in the model at all, and you cannot rely on the confidence to flag all of them.

You can run the same kind of test on your own data without special tools:

1. Hold back your newest rows. When you export labeled rows to train, keep the most recent month or quarter out of the training file.
2. Train on the older rows in StayCharted AI Model Trainer (AMT). AMT tests the model on held-out rows and reports accuracy, confusions and the confidence table.
3. Fill the newest rows you held back, as a CSV or Excel file, and compare the model's answers with the categories your team already gave them. If accuracy on those rows is close to the test report, drift is small. If it is much lower, plan to retrain more often.
4. Count new categories. Check what share of your newest rows have a category that does not appear in the training file. Those rows set a ceiling on what any trained model can do until it has examples.
5. Watch the unsure share over time. If more answers fall below your cutoff than when you started, the incoming rows may be drifting away from what the model learned, and a drift check on your newest rows is worth running.

Retrain when one of these happens:

• You add a category. The model cannot predict it until it has examples. Label some rows, add them, and train a new version.
• Reviewers have built up corrections. Unsure answers from fills, the API, Google Sheets, AI assistants, warehouse fills, the mobile app and the Try box go to one Review Queue. Reviewers confirm or correct them, and the confirmed answers train the next version when a Builder or Reviewer retrains. The Review Queue is on the Essentials and Business plans.
• Your source changes. A new product line, a new form or a new channel can change how rows are written.
• Your drift check shows a gap between the test report and accuracy on your newest rows.

Each new version gets its own test report and confidence table, so check both before you publish it. If the confidence table changed, revisit your cutoff.

Both benchmarks are single runs on public data over one time window each. They show performance on later records, not a controlled estimate of drift or how fast your own data will change. A team whose categories rarely change and whose rows are written the same way every year may see very little drift. A team that adds products or codes every month will see more. The only reliable answer comes from testing on your own newest rows. For every result, see all benchmarks; for plans, see pricing; for features, see the product page.