STAYCHARTED BLOG · TRAINING DATA
How many labeled examples do you need to train an AI classifier?
Plan labeled training data for AI classification. Learn how category balance, ambiguity, duplicates and held-out testing affect the examples you need.
Count examples per category, not only total rows
A thousand rows can still leave a model unprepared if almost all belong to one category. The useful question is how much distinct evidence it has for each decision it must make.
As a practical starting target for a StayCharted pilot, gather around 20 or more reviewed examples per category, and expect to need more for similar or varied categories. This is a collection target, not an accuracy guarantee or a universal minimum.
For ten categories, 20 examples each means 200 rows to begin investigating. It does not mean 200 rows will produce a production-ready model. Training and evaluation both need adequate representation, and small test groups give unstable estimates.
A task with “Billing” and “Account Access” may need fewer examples than one separating 77 closely related banking intents. The wording variety and category boundaries matter as much as the label count.
What makes another example useful?
| New example | Likely value | Why |
|---|---|---|
| A different wording for a familiar request | Useful | Broadens language coverage |
| A rare but important category | Useful | Fills a gap in the data |
| A reviewed borderline case | Useful | Clarifies the boundary |
| The same record copied many times | Limited | Adds little independent evidence |
| A record with a disputed label | Resolve first | Teaches inconsistency otherwise |
For a support model, collect messages that mention both a refund and a login problem if your team has a rule for which intent wins. Clear examples of that rule may help more than another hundred straightforward password resets.
Write a short labeling guide and ask two people to review difficult cases. When people disagree, determine whether the categories need clarification before asking the model to learn them.
Keep an honest evaluation set
Do not judge readiness by accuracy on the same rows used to train. Reserve examples the model never learned from and compare its answers with reviewed labels. Google's dataset-splitting guidance explains why a separate test set is needed.
Keep near-duplicate records and related conversation variants from creating misleading overlap. For recurring workflows, also try a later batch: it tests changes in language, customers and business conditions.
Use validation results to choose settings and review cutoffs. Once you repeatedly inspect and adjust against a test set, it is no longer a clean final check. Set aside another batch for that final assessment.
StayCharted reports results on held-out examples. Read the category-level errors alongside the overall score, and consider how many examples each number represents.
Collect the next batch from the model’s mistakes
- Train an initial model on reviewed examples.
- Find categories with frequent misses or few examples.
- Separate model errors from incorrect labels and missing input information.
- Gather varied, reviewed examples for those gaps.
- Retrain and compare with a stable evaluation set, then confirm on fresh data.
If the model confuses Software and Hardware because every description says only “support,” more copies of that text may not solve the problem. Add available context or revise the task. Data quantity cannot compensate for an input that omits the evidence needed to decide.
If you have no labeled history, paid-plan grouping can help propose starting groups, but a person still needs to name, review and correct them. Machine-proposed groups are not automatically trustworthy answer labels.
Know when you have enough to start
You have enough for a useful pilot when each intended category has representative examples and you can measure where the model fails. You have enough for a particular deployment only when fresh-batch results and the required human review fit that workflow's risks and capacity.
Keep collecting corrections after launch. New products, teams or policies can change the task. Retraining should follow reviewed changes in the data, not an assumption that the initial model remains suitable indefinitely.
See training best practices and how to choose a review cutoff before moving from a pilot to routine use.
Common questions
Is 100 examples enough for text classification?
It can support an initial experiment with a small, clear category set, but it is not a guarantee. Consider examples per category, wording variety and the size of the independent evaluation set.
Is 20 examples per category a product guarantee?
No. It is a practical starting collection target. Similar categories, rare cases and diverse language often require more examples and careful evaluation.
Should I duplicate rare-category rows?
Copying identical rows does not add independent evidence and can distort evaluation. Prefer additional distinct, reviewed examples and inspect category-level results.
Can I train without labeled examples?
Supervised classification needs answer labels. Grouping can help organize unlabeled rows on paid plans, but a person must review and name useful categories before relying on them.