How many categories for AI classification: what our benchmarks show as the list grows

Each row below is from a published benchmark page, measured on held-out test rows. The datasets differ, so the rows are not directly comparable with each other; the point is the spread.

Three things stand out. Accuracy did not fall in a straight line as categories grew. The trained models led prompting by the widest margin on 72 close subsystems. And on the shortest list, the trained models did not win.

The clearest test is NHTSA, where the same complaints were classified at two levels of detail: 21 vehicle systems and 72 subsystems. Going from 21 to 72 categories:

• the AI Classifier Model went from 84.1% to 76.9%;
• the Dedicated AI Model went from 87.7% to 85.9%;
• Snowflake AI_CLASSIFY went from 82.6% to 52.7%;
• Databricks ai_classify went from 83.3% to 56.1%.

The two warehouse functions read category names and a one-line task description. On 21 broad systems that was nearly enough. On 72 close subsystems it was not, and the Snowflake and Databricks comparison sums it up: on 72 close categories the two functions "filed 53% and 56% correctly." The trained models had learned the boundaries from labeled complaints. (The two levels used different training windows, 2023 to 2024 for systems and 2020 to 2024 for subsystems, so the trained models' drop is not a perfectly matched comparison.)

The errors on other benchmarks show what "close" means in practice. On BANKING77, the prompted model labeled 32 of 40 get_physical_card messages as change_pin. On LEDGAR it labeled 34 "Waivers" clauses as "No Waivers" and 34 "Definitions" clauses as "Defined Terms." Those categories have names that overlap and boundaries set by the people who built the dataset. The benchmark page calls this a plausible explanation, not an isolated cause, and we treat it the same way. Your own taxonomy almost certainly has pairs like these.

A long list becomes hard when most categories have only a handful of examples. The CBP benchmark is the extreme case: 5,621 training rulings covered 2,014 distinct 10-digit codes, and 1,129 of those codes were seen only once. On top of that, 27% of test rulings had a code that never appeared in training. Exact-code accuracy was 22.2% and 25.1%.

The same models were far more accurate at coarser levels of the same codes:

The photo benchmarks show the same effect inside one list. On car damage types, the Dedicated AI Image Model scored 70.2% overall on 8 types: 77.7% on the four common types and 19.0% on the four rare ones. Rare categories are where a model is weakest, whatever the total count.

WANDS is the counterexample. Its 158 product categories are fine-grained, but most are easy to tell apart from a product's name and description, and training had 11,370 products. The product categorization results were 95.3% for the AI Classifier Model and 97.3% for the Dedicated AI Model, with the right department 96.9% and 98.4% of the time. A long list is fine when each category is distinct and well supplied with examples.

The test report AMT produces for every trained model shows accuracy per category and the confusions between them. Use it to judge your taxonomy size:

• Confused pairs. Two categories that the model mixes up in both directions often overlap in meaning, not just in wording.
• Thin categories. Categories with few examples tend to score lowest, as the rare damage types did.
• A broad catch-all. On Medical Abstracts, the prompted model scored 40.2% on the broad "General pathological conditions" group. A category that means "everything else" is hard for any model.
• Few broad categories. With 5 groups, the trained models scored 60.0% and 63.6% against the prompted model's 65.7%. The page says that gap is not established as a real difference. If your list is short and broad, a trained model may not be worth the setup, and prompting with good definitions can be enough.

Merge categories that overlap

If the business does not act differently on two categories, merge them. On the damage benchmark, the Dedicated AI Image Model named the exact exterior part on 89.9% of test pictures, and 93.7% if front and back wheels counted as one. Only merge when the distinction does not matter to anyone downstream; when it does, collect more examples of both instead.

Collect examples per category, not just in total

A big file can still leave some categories thin. Count examples per category and add more for categories that look alike. Prompting has the same arithmetic: the assistant benchmark notes that five examples across 77 or 100 categories means 385 or 500 examples in every prompt. A trained model learns them once. For guidance on numbers, see how many examples you need to train a classifier.

Use a two-level taxonomy

Split a long list into a short top level and a detailed second level, the way HS codes split into chapters and NHTSA complaints into systems and subsystems. Train one model on the top-level column and another on the detailed column. Use the top-level answer where the detailed model is unsure: in the CBP test the Dedicated AI Model was right on the chapter 74.0% of the time even though it matched the exact code only 25.1% of the time. Detailed answers below your cutoff go to the Review Queue (Essentials and Business plans), and reviewers' answers train the next version.

Write down what each category means

Most overlapping categories exist because nobody wrote down where one ends and the next begins. Why taxonomy matters covers definitions, borderline decisions and reviewer agreement.

Starting without labels

If you have rows but no categories yet, AMT can start from unlabelled rows (Essentials and Business). Let it group similar rows and name the groups you keep, or give it your category names and let it suggest one for every row. It starts with as few groups as the rows support, and finer categories come when you ask for them or make them yourself. Groups you give the same name merge. People review the suggestions before training. We have not benchmarked this step, so we quote no accuracy for it.

To try your own list, start free. See all benchmarks, pricing and the product page.