Machine learning for support ticket classification: train on the tickets your team already resolved

A support ticket classification model reads the text of a ticket (usually the subject and description, sometimes the channel or product field too) and picks one category from your list. That list can be queues ("Billing", "Access requests", "Hardware"), intents ("card not arrived", "refund status") or assignment groups. Instead of writing rules for it, you give it examples.

In StayCharted AI Model Trainer (AMT), the steps are:

• You upload rows your team has already categorized: ticket text plus the category it ended up in.
• AMT checks the data, runs personal-information checks and masking, and trains a private model.
• It tests the model on held-out tickets it never saw during training and reports accuracy overall and per category, which categories it confuses, and how many answers fall above each confidence cutoff (and how many of those are right).
• New tickets come back with a category, a confidence from 0 to 1 and a needs-review flag.

Many queues that overlap

Most support teams end up with dozens of categories, and the hard ones sit next to each other. "Can't log in" and "account locked" read almost the same. So do "card not arrived" and "card not working". Keyword rules handle the first few obvious queues and then break down, because the words that separate two close categories are often the same words.

Your rules were never written down

Every team has unwritten rules: VPN tickets from a certain office go to network, not desktop support, or anything that mentions a chargeback goes to disputes even if it starts as a refund question. A general AI model only knows your category names and whatever definitions you write for it. A model trained on resolved tickets learns those rules from the decisions your agents already made, including the reassignments.

The same ticket should get the same answer

Routing has to be predictable. A trained classifier is a fixed model, so a given ticket gets the same answer until you train a new version. A prompted assistant can vary. In our benchmark, asking Claude the same question twice gave the same category on 99.2% of the first 500 banking messages, which is close, but it is agreement between two attempts, not accuracy.

We tested this on BANKING77, a public dataset of customer support messages labeled with 77 intents. It is a good stand-in for a support queue: short messages, many categories and several pairs that are easy to mix up. Every model was scored on the same 3,080 held-out test messages. The trained models learned from 10,003 labeled training messages. Claude Sonnet 5.5 received the category names and one-line definitions, with no labeled examples. Full method in the assistant vs. trained model benchmark and the BANKING77 report.

The Dedicated AI Model was 9.0 points ahead of the prompted assistant. Even the Classifier Model, which uses no AI at all, came out ahead of the assistant once it had seen labeled examples.

The mistakes show why. Claude labeled 32 of 40 "get physical card" messages as "change PIN". The category names alone did not tell it how this bank's customers phrase that request. The trained models had seen hundreds of real examples of each intent, so they learned where the line between close categories actually falls.

Keep the limits in mind. This is one public dataset of short banking messages, not your tickets, and the margins on these figures are roughly ±0.9 to ±1.8 points. Your own accuracy will depend on your categories and how consistently your team labeled past tickets, which is why AMT tests every model on your held-out rows before you use it.

Accuracy alone does not tell you how much work the model takes off your agents. The useful number is how many tickets can go straight through and how many need a person. Each answer has a confidence, and you choose a cutoff: answers above it route automatically, answers below it go to the Review Queue.

Here is how cutoffs behaved for the Dedicated AI Model on the 3,080 BANKING77 test messages:

At a 0.90 cutoff, a person would review about 9 in 100 tickets, and those tickets contained about two thirds of the model's mistakes. Our pricing page puts this in monthly terms: across 10,000 support messages, a person checks about 2,800 rows with the Classifier Model, about 1,800 with the AI Classifier Model and about 900 with the Dedicated AI Model to end at roughly 98% right.

These are projections, and they come with caveats. They assume the reviewer corrects every flagged ticket, no real reviewers took part in the test, and the cutoffs were chosen after seeing the test results. Confident mistakes still pass through: at 0.90, about a third of the errors were above the cutoff. Start with a cautious cutoff on your own data and raise the share that goes straight through as you see how the model does. More on this in AI confidence scores and human review.

Reviewed tickets also feed the model. The answers your agents give in the Review Queue become training rows for the next version of the model, so it improves on exactly the tickets it found hard.

Training a model is worth it when you have many categories, close categories or rules only your team knows. It is often not needed when:

• You have a handful of broad queues (for example "Sales", "Support", "Billing") that keyword rules or your helpdesk's own triggers already sort well.
• A wrong route costs little because agents reassign tickets quickly anyway.
• You have no history yet. A model needs examples; if you have none, start by labeling as you go.

The benchmark shows this too. On a separate dataset of medical abstracts with only 5 broad condition groups, Claude given definitions scored 65.7% and the Dedicated AI Model scored 63.6%, so the trained model did not win there (the gap is not yet shown to be a real difference). Fewer, broader categories leave less for training to add. Support queues with dozens of overlapping categories are the opposite case.

AMT trains three kinds of text model, and you can compare them on the same held-out tickets:

• Classifier Model: learns words and phrases, no AI. Fast and cheap to run, on every plan. A good baseline when tickets use consistent wording.
• AI Classifier Model: a ready-made AI language model reads the meaning and a private classifier learns your categories. On every plan, and the Free plan includes one. A good default for intent classification in customer support.
• Dedicated AI Model: an AI language model fine-tuned on your own tickets. The most accurate in the BANKING77 test and the fewest tickets to review. On the Business plan.

Plans are Free ($0), Essentials ($49) and Business ($99) per workspace per month. See pricing and the product overview.

1. Export resolved tickets. A CSV or Excel file with the ticket text and the final category or queue. Use the category the ticket was resolved under, not the one it was first assigned.
2. Upload and train. AMT checks the data, masks personal information if you choose, trains the model and tests it on held-out tickets.
3. Read the report. Look at accuracy per category and the confusions. A pair of categories the model mixes up is often a pair your agents mix up too.
4. Pick a cutoff and connect. Fill files or a Google Sheet to try it, then call the model from the API (Essentials and up), the Power Automate connector, Claude or ChatGPT, or Snowflake and Databricks tables. Low-confidence tickets land in the Review Queue.

For setup inside a specific helpdesk, see Zendesk ticket classification with AI and how to auto-assign ServiceNow incidents. For an overview of routing, see AI support ticket classification, and for banks and fintechs, banking customer message classification.

You can start free with one AI Classifier Model and a file of tickets your team has already sorted.

The Review Queue requires Essentials or Business. Retrain and publish a new version to use reviewer corrections in future answers.