Benchmark · Automotive · Quality and safety · Text
Can AI file vehicle complaints where NHTSA filed them? We measured it on 5,000 complaints from 2025 onward
StayCharted classified 5,000 later NHTSA complaints: 87.7% system and 85.2% subsystem accuracy. Compare models, confidence, mistakes and the test method.
At a glance
| StayChartedAI Model Trainer · model comparison | ||
|---|---|---|
| 5,000 complaints NHTSA received from January 2025 on | Vehicle system (21 categories) | Subsystem (70 categories) |
| Always guessing the most common category | 24.6% | 31.8% |
| StayCharted AI Classifier Model | 84.1% | 76.9% |
| Dedicated AI Model | 87.7% | 85.2% |
| StayCharted Dedicated, confidence 0.90 or higher: share of complaints | 84.7% | 70.9% |
| Right among those | 93.7% | 94.4% |
The data
NHTSA's Office of Defects Investigation publishes every consumer complaint it receives, with the vehicle, the date received, the complaint narrative and the component it was filed under. The public NHTSA complaint dataset is our source. We publish results only, never complaint text.
From the 2020 to 2024 and 2025 to 2026 complaint files we kept:
- Vehicle complaints only, not tires or equipment sold separately.
- Complaints filed under exactly one component. A complaint filed under several has several right answers, so we left it out rather than guess. "Unknown or other" was left out too.
- Narratives of at least 40 characters, and each exact narrative once.
NHTSA renamed some systems over the years, for example "Service brakes, hydraulic" and "Service brakes, electric" became "Service brakes". We merged these vehicle-system names. Some subsystem variants remain separate, including “Service brakes: antilock” and “Service brakes: antilock/traction control/electronic limited slip”; choosing the other name is counted as wrong.
Two levels, split by time
| Vehicle system | Subsystem | |
|---|---|---|
| Example | SERVICE BRAKES | ELECTRICAL SYSTEM:INSTRUMENT CLUSTER/PANEL |
| Categories | 21 | 72 in training, 70 of them in the test |
| Training | 19,902 complaints received 2023 to 2024 | 19,281 complaints received 2020 to 2024 |
| Test | 5,000 complaints received from 1 January 2025 | 5,000 complaints received from 1 January 2025 |
The split is by date. The models learned from older complaints and were tested on newer ones, the way a quality team would use them. That's a harder and more honest test than a random split: vehicles, features and the way people describe them change from year to year.
How we tested
We uploaded each training file to StayCharted AI Model Trainer and trained two models at each level, with default settings and no hand tuning:
- StayCharted AI Classifier Model: fits meaning-only and words-plus-meaning candidates, keeping the one that scores better on held-out validation examples.
- StayCharted Dedicated AI Model: a language model fine-tuned on the same examples, on a GPU. It learns the categories from the same labeled examples.
Then we filled the test file with each published model, the way a customer fills a spreadsheet, and compared every answer with NHTSA's filing. Every answer comes with a confidence, so we could also measure what share of complaints a model is sure of and how often it's right on those.
Results
Accuracy
| StayChartedAI Model Trainer · model comparison | ||
|---|---|---|
| Vehicle system | Subsystem | |
| Always guessing the most common category | 24.6% (Engine) | 31.8% (Structure: body) |
| StayCharted AI Classifier Model | 84.1% | 76.9% |
| Dedicated AI Model | 87.7% | 85.2% |
Which answers can go straight through
The confidence is what lets a team act on the answers. A team can route answers above a chosen cutoff into its records and send the rest to a person. Confident answers can still be wrong; choose the cutoff for your task and sample them too.
Vehicle system:
| Confidence at least | StayCharted AI Classifier: share, right among them | StayCharted Dedicated: share, right among them |
|---|---|---|
| 0.80 | 66.8%, 95.6% | 89.8%, 92.3% |
| 0.90 | 54.7%, 96.7% | 84.7%, 93.7% |
| 0.95 | 42.6%, 97.2% | 79.2%, 95.2% |
Subsystem:
| Confidence at least | StayCharted AI Classifier: share, right among them | StayCharted Dedicated: share, right among them |
|---|---|---|
| 0.80 | 51.6%, 92.2% | 81.1%, 92.4% |
| 0.90 | 39.0%, 94.8% | 70.9%, 94.4% |
| 0.95 | 29.6%, 95.6% | 62.1%, 95.7% |
At the 0.95 cutoff both models are right about 95% of the time on the answers they keep. The StayCharted Dedicated AI Model keeps twice as many of them at the subsystem level: 62.1% of complaints against 29.6%. For a team, that's the difference between a person reading four complaints in ten and one reading seven in ten.
Where it does well, and where it doesn't
Among the larger subsystems, the StayCharted Dedicated AI Model filed these where NHTSA did:
| Subsystem | Test complaints | StayCharted AI Classifier | StayCharted Dedicated |
|---|---|---|---|
| Structure: body | 1,592 | 95.9% | 93.0% |
| Electrical system: instrument cluster/panel | 532 | 52.4% | 98.1% |
| Back over prevention: sensing system | 379 | 93.9% | 94.2% |
| Forward collision avoidance: automatic emergency braking | 336 | 90.8% | 89.0% |
| Forward collision avoidance: adaptive cruise control | 161 | 70.8% | 82.6% |
| Power train: automatic transmission | 121 | 90.1% | 95.0% |
| Forward collision avoidance: warnings | 125 | 42.4% | 56.0% |
| Latches/locks/linkages: doors | 88 | 47.7% | 59.1% |
The weak spots are neighbors. "Warnings" against "automatic emergency braking", door latches against the body, rear-view braking against the sensing system: a driver describing a collision warning often describes the braking that followed it. When the StayCharted Dedicated AI Model got the subsystem wrong, a third of the time (245 of 742) it still had the right vehicle system.
Most frequent confusions, StayCharted Dedicated AI Model, subsystem:
| NHTSA filed it under | The model answered | Complaints |
|---|---|---|
| Back over prevention: rearview system braking | Back over prevention: sensing system | 38 |
| Forward collision avoidance: warnings | Forward collision avoidance: automatic emergency braking | 38 |
| Latches/locks/linkages: doors | Structure: body | 33 |
| Visibility: glass, side/rear | Structure: body | 31 |
| Structure: body | Visibility: sun/moon roof assembly | 26 |
Newer complaints are harder
When StayCharted trained these models, it tested each one on complaints held back from the training years. Three of those scores were higher than on complaints from 2025 onward:
| Held back from the training years | Complaints from 2025 on | |
|---|---|---|
| Vehicle system, AI Classifier | 85.9% | 84.1% |
| Vehicle system, Dedicated | 88.6% | 87.7% |
| Subsystem, AI Classifier | 80.3% | 76.9% |
| Subsystem, Dedicated | 83.7% | 85.2% |
Three of the four dropped, by up to 3.4 points. New vehicles and features bring new ways of describing problems, which is why the unsure answers go to a person and their decisions train the next version.
What this means for your own data
NHTSA complaints are written by drivers, often in a hurry, and filed by a component list built for a regulator. Your warranty claims and technician lines are written by people who know the vehicle, and filed by codes your engineers chose. This test did not measure warranty claims or technician repair-order lines. Your own report may differ in either direction, especially where codes overlap. Run it on your own coded cases: the report shows accuracy for each of your codes, which ones it confuses, and the cutoff table above for your own data.
Limits of this test
- NHTSA's filing is the answer key. Some complaints could fairly go under two components, and the test counts only NHTSA's choice as right.
- Blank answers. The StayCharted Dedicated AI Model left 47 subsystem complaints blank, most of them battery complaints, whose category name, "Electrical system: 12V/24V/48V battery", is long and full of numbers. Blank answers are counted as wrong.
- One run each, with default settings and a fixed random sample (seed 17). No tuning on the test.
- Text only. No vehicle make, model or year was given to the models.
Use the findings in your workflow
Frequently asked questions
Can AI classify vehicle complaints by component?
Yes. On 5,000 NHTSA complaints from 2025 onward, a model trained on earlier complaints filed 87.7% under the same vehicle system as NHTSA and 85.2% under the same subsystem, from the narrative alone.
Which answers can be trusted without a person checking?
At confidence 0.90 or higher, the StayCharted Dedicated AI Model covered 70.9% of subsystem test complaints and was correct on 94.4% of those. That is measured performance, not a guarantee. Choose a cutoff using your own test and the cost of errors; sample confident answers too.
Can it use our own failure codes?
Yes. The benchmark uses NHTSA's components because they're public. A model learns whatever codes your team used in its training examples.
Does it work on warranty claims and repair orders?
The same approach applies to any text filed under a code: warranty claim narratives, technician corrections, call-center notes and field reports. Test it on your own coded cases first.
How was the test kept honest?
The models learned from complaints received up to the end of 2024 and were tested only on complaints received from 2025 on, which they never saw.