Machine learning for classification: when a model trained on your own decisions beats rules and LLM prompts

Rules and keywords. "If the ticket mentions refund, send it to Billing." Rules are cheap and easy to read, and they work for a handful of clear categories. They break when people describe the same thing in different words, they start to conflict as they pile up, and someone has to maintain them.

Prompting a general LLM. You give an AI assistant the category names, perhaps a definition of each, and ask it to pick one. No training data is needed, and on broad, self-explanatory categories it does well. It only knows what fits in the prompt, so the conventions your team follows without writing them down never reach it.

A model trained on your own decisions. The model learns from thousands of rows your team has already categorized, including the exceptions and the close calls. It is tested on rows it never saw, so you know its accuracy before you rely on it, and it gives every answer a confidence, so the sure answers can go straight through and the unsure ones go to a person.

In StayCharted AI Model Trainer (AMT), that third option comes in several kinds: a Classifier Model (no AI) that learns words and phrases, an AI Classifier Model where a ready-made AI language model reads meaning and a private classifier learns your categories, and a Dedicated AI Model where the AI language model itself is fine-tuned on your examples. Photos get the same choice with the AI Image Classifier Model and the Dedicated AI Image Model.

In the text tests below, Claude Sonnet 5.5 received a definition of each category, and the Snowflake and Databricks functions received the category names (with a short task description on the subsystem runs). The StayCharted models learned from labeled examples: about 10,000 rows for BANKING77, LEDGAR, Medical Abstracts and each search relevance set, and about 19,000 to 20,000 earlier complaints for each NHTSA level. Every score is on held-out rows the models never saw. Bold marks the best score in each row.

The image tests compare two ways of training on your own labeled photos: a ready-made vision AI with a classifier trained on your categories, and a vision model fine-tuned on the photos themselves. No general LLM was tested on the photos.

Results across different datasets aren't directly comparable. Each comes from one setup on one dataset, and none of them guarantees the same result on your data.

Many categories, or close ones

The clearest case is the NHTSA test. On 21 broad vehicle systems, the warehouse functions were close to the trained models. On 72 subsystems (70 of them in the test), Snowflake AI_CLASSIFY filed 52.7% of 5,000 complaints where NHTSA filed them, Databricks ai_classify 56.1%, and the Dedicated AI Model 85.9%. That is about 1,500 to 1,650 more complaints in the right place. Names like "Back over prevention: sensing system" and "Back over prevention: rearview system braking" are hard to tell apart from the names alone and easier for a model that has learned from thousands of earlier filed complaints. More in machine learning for warranty claim classification.

Close categories trip up a prompt even when the list is manageable. On BANKING77, Claude labeled 32 of 40 get_physical_card messages as change_pin.

Your own categories, with rules nobody wrote down

On LEDGAR, Claude labeled 34 "Waivers" clauses as "No Waivers" and 34 "Definitions" clauses as "Defined Terms". Both readings are defensible in plain English, and both are wrong under the dataset's conventions. Labeled examples carry those conventions; a category name can't. The trained model's lead was largest on tasks like this (9.0 points on BANKING77, 8.0 on LEDGAR), although the comparison doesn't isolate the cause. Search relevance shows the same effect: on WANDS, the Dedicated AI Model reached 80.2% against Claude's 72.7%, because the relevance labels follow the judgment of the people who labeled the data.

A confidence you can trust

Accuracy tells you how often the model is right. Confidence tells you which answers to trust, which decides how much work goes straight through. On the 72 NHTSA subsystems, the Dedicated AI Model was at 0.90 confidence or higher on 71.5% of complaints and right on 94.3% of those. Databricks ai_classify also returns a confidence, but on the same subsystems it reached 0.90 on only 2.8% of complaints and was right on 62.3% of them, so almost every answer would still need a person. How to use confidence scores with human review →

The same answer every time, at volume

When Claude was asked the same 500 rows twice, it gave the same answer 97.0% to 99.2% of the time, depending on the dataset. That is high, but a classification that drives a report, a claim or an audit should give the same answer for the same input. A prompt also carries the category list and instructions on every single row, while a trained model has learned the categories already. We didn't measure cost in these tests, so we don't quote a figure, but the difference grows with the number of rows and categories.

A trained model isn't always the answer. In four sets of results it lost, tied or led only narrowly, and each one says something about when training is needed.

• Medical Abstracts (5 categories). Claude scored 65.7%, the Dedicated AI Model 63.6% and the AI Classifier Model 60.0%. The categories overlap, and the hardest one was hard for everyone: Claude scored 40.2% on "General pathological conditions". The margins (about 1.7 to 1.8 points either way) overlap, so we read this as close rather than a clear win either way. Medical abstract classification →
• Amazon ESCI search relevance (4 labels). Claude scored 59.9% and the Dedicated AI Model 59.0%, a small difference in one run. Its four answers are fine distinctions, and some labels appear inconsistent, which limits what any model can learn from them.
• Part numbers from a photo (100 look-alike parts). The AI Image Classifier Model named the exact part on 87.8% of 1,000 test photos, and the Dedicated AI Image Model reached 75.3%. There were only 20 training photos per part. Our reading is that fine-tuning a vision model needs more examples than that, while the ready-made vision AI already tells fine shapes apart. The lighter trained model won here; training itself still did the work of learning the 100 parts.
• Ties and near ties. On FruitNet fruit grading, both image models scored 98.7%. On the 21 NHTSA vehicle systems, Snowflake AI_CLASSIFY with a task description and one example per category reached 84.2%, level with the AI Classifier Model's 84.1% and 3.5 points behind the Dedicated AI Model.

The pattern across these tests is consistent. Training on your own decisions pays off when the categories are many, close or your company's own, and when you have the labeled history to learn from. A short list of broad, self-explanatory categories may not need it.

Train a model when:

• You have dozens to thousands of categories, or several that differ by a detail (subsystems, clause types, part numbers, tariff codes).
• The categories follow conventions your team applies without writing them down.
• You have history: rows your team has already categorized. The text tests here used about 10,000 to 20,000; how many examples you need covers smaller sets.
• You want part of the work to go straight through, which needs a confidence you can check against held-out rows.
• The same input must get the same answer, for reporting or audit.

Keep it simple when:

• There are a few broad categories whose names say what they mean, and an occasional wrong answer is cheap.
• You have no labeled history yet. Start with a prompt or with Your AI Model in AMT (your own LLM given your categories and similar past rows, not trained), let reviewers correct the answers, and train once the corrections add up.

Start with the lighter model. The part-photo test shows a bigger model isn't automatically better, and the AI Classifier Model runs on every plan.

Each post below covers one use case: what goes wrong with rules and prompts there, what we measured, and how to set it up. The figure beside each link is the headline result from that benchmark.

Text classification machine learning

• Warranty claims and vehicle complaints: 85.9% on 72 NHTSA subsystems with the Dedicated AI Model, against 52.7% and 56.1% for the warehouse functions.
• Support tickets: 93.7% on 77 BANKING77 intents, against 84.7% for Claude.
• Contract clauses: 84.9% on 100 LEDGAR clause types, against 76.9% for Claude.
• Product categorization: 97.3% on 158 Wayfair WANDS product categories.
• Search relevance: 80.2% on WANDS against Claude's 72.7%, and a near tie on Amazon ESCI.
• HS codes: our hardest test, with 39.2% at the 6-digit subheading and 25.1% on the exact 10 digits out of 2,014 codes, from a one-line description. A job for suggestions with review, not straight-through filing.
• Access requests and permissions: on group access, 97.8% with the Classifier Model (no AI) against 97.6% for a same-as-most-of-the-team rule; on access requests, it ranked the ones to deny far better than the best rule (AUC 0.84 against 0.70).
• Spend classification: a step-by-step guide to categorizing spend from your own reviewed history.

Image classification machine learning

• Vehicle damage: 86.4% on 6 damage categories with the Dedicated AI Image Model, against 76.6%, and 70.2% against 64.1% on 8 damage types.
• Part identification from photos: 87.8% on 100 look-alike parts, where the AI Image Classifier Model beat the Dedicated one.
• Food quality inspection: 98.7% on FruitNet grades with both models, and on cashews 93.8% against 90.0%, with the Dedicated AI Image Model finding 26 of 40 defects against 16.

1. Export rows your team has already categorized, as a CSV, Excel or JSON file, a Google Sheet, or a Snowflake or Databricks table. Recent rows are best.
2. Upload them to AMT. It checks the data and can find and mask personal information before training, on every plan.
3. Train and read the test report. It shows accuracy overall and per category, the confusions, and the share of answers above each confidence cutoff with how many of those are right.
4. Compare it with what you use now, on the same held-out rows. If a prompt or a rule already does as well, keep it.
5. Fill new rows in files, Google Sheets, Power Automate, Claude or ChatGPT, through the API, or straight into a results table in your warehouse. Answers below your cutoff go to the Review Queue, and your reviewers' answers train the next version.

The Free plan includes one AI Classifier Model. The Dedicated AI Model is on Business. See plans → · How the product works → · Start free →

The Review Queue requires Essentials or Business. Retrain and publish a new version to use reviewer corrections in future answers.