Best practices: getting a model your team can trust

A good first job:

• Happens often. Hundreds of times a month is ideal; a model pays off on repetition.
• Has clear answers. Someone on your team could say whether an answer is right.
• Has history. You already have past items with the answer your team gave.
• Can be decided from what's in the row. If a person needs a phone call or a document that isn't in the data, the model won't know it either.

Start small: one queue, one department, one request type. Measure it, then expand.

One row per item, with the answer your team gave. A spreadsheet works: one or more columns describing the item, and one column with the answer.

Enough examples of each answer. About 20 per category is a start; a few dozen is better. With only a handful of examples, category performance is harder to judge reliably. These counts are starting points, not a guarantee; test on representative held-out examples. If a category is rare but important, add more of it rather than more of everything else.

Consistent labels. If two people would file the same row differently, the model learns both answers.

• Write one line per category saying what belongs in it (the model's labeling guide).
• Merge categories that mean the same thing ("Billing" and "Invoices").
• Split or rename categories nobody can tell apart.

Only what's known at decision time. Leave out columns filled in after the decision, such as resolution notes, "closed by" or the final cost. They make the test score look great and the model useless on new items.

Current decisions only. Remove examples from before a policy changed, and access or codes that were later reversed. The model repeats what it's shown, including old mistakes.

Fix what Data quality points out. Before training, StayCharted flags duplicate rows, categories with too few examples, and supported personal-information patterns such as email addresses and phone numbers. These checks do not find all personal information. Remove, merge or mask them, each in a click.

No answers yet? On Essentials and Business, Start from rows with no categories groups similar rows for you to name, so you can build the first examples quickly.

Read how many labeled examples you need to plan coverage for each category.

Don't guess: train two kinds on the same examples and compare their reports. The cheaper model is often good enough.

Several answers at once (for example GL account and cost center, or a list of groups for a file) can be one model. Choose the answer's shape when you map your columns.

Every version is tested on examples it never saw during training.

• Accuracy by category comes first. An overall 95% can hide a category that's right only half the time. Read the categories that matter most to you first: the defect, the escalation, the access that shouldn't be given.
• Look at the mistakes. "Where the model got it wrong" often shows rows that were in the wrong category in your file. Fix those labels and retrain; that's usually the fastest improvement.
• Look at what it confuses. Two categories that are always mixed up need a clearer definition, a column that tells them apart, or merging.

Every answer has a confidence. On Essentials and Business, answers below your review cutoff enter the Review Queue. You can also route them to a person in your own workflow.

• The report shows, for each cutoff, how many answers a person would check and how many of the model's mistakes those checks would catch.
• Choose by what a mistake costs. Routing a ticket to the wrong queue is cheap; giving access to the wrong person or missing a defect is not. Higher stakes, higher cutoff.
• Choose by what your team can check. A cutoff that sends half the work back to people saves little. Find the point where checking a small share catches most mistakes.
• Confident answers can still be wrong. Spot-check a sample of them now and then, especially early on.

See choosing a confidence threshold for human review for a practical way to balance errors and review effort.

• Fill a file: upload new rows with the answer column blank. Every row gets an answer, a confidence, and whether it needs review.
• Google Sheets: fill empty answer cells in your sheet with the Google Sheets add-on.
• API (Essentials and up): call the model from Power Automate, ServiceNow, Zapier or your own code.
• AI assistants: ask your model from Claude or ChatGPT.

Send the same columns the model learned from. A model trained on "Subject" and "Body" needs "Subject" and "Body", not just one of them. Fields it doesn't read are ignored.

Decide what happens to unsure answers before you go live. Who checks them, and where: the Review Queue, your ticketing tool, or an approval step. Check each item in one place, not two.

• Review Queue (Essentials and Business): unsure answers wait for a person. Confirm or correct them; repeated items are grouped, so one decision settles them all.
• Every correction is a new example. Nothing changes until you retrain, so a wrong click doesn't change the model on its own.
• Retrain on a rhythm: monthly, or when the review pile shows a pattern. Add new categories as soon as they appear in your work.
• Publish only better versions. Compare the new report with the live one, category by category, before you publish. The previous versions stay listed.

• Suggest, don't decide, for anything sensitive: access to files and systems, money, safety, anything regulated. Use the model to prepare and order the work; keep a person's approval as the decision.
• Anything that widens access gets an approval. That holds for SharePoint, NTFS or request workflows, however confident the model is.
• Clean history after audits and access reviews. Remove what was reversed, so the model stops learning it.
• Mask personal information you don't need for the decision. The model rarely needs a name or an email address to choose a category.

• One model per decision. "Which queue" and "how urgent" are two models, or one model with a two-part answer. They aren't one mixed list of labels.
• Use roles: Builders train and publish, Reviewers check answers and can start a retrain from reviewed corrections, and Viewers read. Publishing stays with a Builder.
• Keep the labeling guide current. It's the shared definition of your categories, for new team members, reviewers and AI assistants alike.
• One workspace per department or customer when their categories or data must stay separate.