Human-in-the-loop machine learning: your team trains the model while they use it

There are three ideas with overlapping names:

• Human in the loop: a person checks some of the model's answers before they're used, and those checks feed back into training.
• Active learning: the model chooses which examples a person should look at, usually the ones it's least sure of, to focus labeling effort on uncertain cases.
• Data flywheel: the effect over time. More use brings more checked examples, which can improve the next model and make it useful in more situations.

In AMT the three are one workflow. The model's confidence decides what a person sees (active learning), the person's answer is the check (human in the loop), and retraining on those answers is the flywheel.

On Essentials and Business, a model in AMT answers through many channels, and all of them feed one place, the Review Queue:

Every answer comes with a confidence from 0 to 1. Each model has a review cutoff, and answers below it go to the Review Queue on their own. The person who asked still gets the answer, marked as unsure: in the app, it says "Sent to your team to check".

Choosing the cutoff. The cutoff is a trade-off between how much your team checks and how right the output is. AMT's test report shows it on rows the model never saw: at each cutoff, the share of answers a person would check, and how many of the model's mistakes are among them. In our part identification benchmark, the AI Image Classifier Model at its default cutoff was sure of 32% of photos and right on 322 of those 323; the rest fell below the cutoff and would need a person to check them.

The Review Queue is one screen for every model in a workspace. A Reviewer, often the taxonomist, parts specialist or senior agent who knows the categories best, works through it least sure first:

• Confirm or correct. Pick the right category, or confirm the model's.
• See what the team decided before. For each item, the queue shows the model's top categories and similar items your team has already labeled, found by meaning, not just matching words.
• Skip or mark as unsure when the item can't be decided.
• Use an assistant if you like. The queue is also open to Claude or ChatGPT through the AMT connector, so a reviewer can work through it in conversation. Answers recorded that way still need the person's yes.

Builders and Reviewers get a daily email listing how many new answers are waiting, per model, so the queue doesn't pile up unseen. When an unsure answer came from a filled file, the file can be downloaded again with the reviewed answers in.

Once reviewed answers are waiting, a Builder or a Reviewer starts a retrain from the queue. AMT adds the reviewed answers to the training data and trains a new version with the same settings as the live one, then tests it on held-out rows like any version. A Builder decides whether to publish it.

Two things keep this honest:

• Every version is measured before anyone uses it: accuracy overall and per category, and the confidence table. You see whether the reviewed answers helped.
• Nothing is retrained for nothing. If every reviewed answer is already in the training data with the same label, no training run is spent.

Over time, the share of answers that needs checking should fall, because the model has learned the cases it used to be unsure of. That's the flywheel. How fast depends on your data. We haven't published a measured curve for it, so we won't quote one.

A loop that learns from people also learns their mistakes. When many people review, some will disagree, and some will be wrong. Three habits help:

1. Give reviewers a labeling guide. A sentence per category, and a rule for each pair that overlaps. Most disagreements come from categories that overlap, not from carelessness.
2. Keep the reviewers few and expert. The Review Queue is for the people who know the categories, not everyone who asks.
3. Check the training data now and then. In our label error benchmark we planted 711 wrong labels in 14,214 products, and two models, each trained on one half and checking the other, flagged 710 of them. Among flags at confidence 0.9 or higher, 95.4% were planted errors. That is a result for this test, not a guarantee for other data. Cross-checking can help find mistakes a review loop lets in.

A maintenance team trains an AI Image Classifier Model on photos of its spare parts and publishes it. Technicians, some with an AMT account and some with an access code, photograph parts on site with the Android app.

• For answers above the team’s cutoff, the technician follows the team’s normal process.
• A photo of an older part in poor light comes back at 41% confidence. The technician sees "Sent to your team to check" and asks the parts desk as before.
• That afternoon, the parts specialist opens the Review Queue, sees the photo with the model's top guesses and similar parts the team already identified, and picks the right number.
• At the end of the week, the specialist retrains. The reviewed photo joins the training examples. A Builder checks the new test report before publishing; later photos may still need review.

No one on site did anything extra to train the model. The photos they were already taking did it.

1. Start free and train a model on the examples your team already has.
2. Read the test report and set the review cutoff where the trade-off suits you.
3. Invite the people who'll review as Reviewers, and connect the channels you use: files, API, Sheets, assistants, a warehouse or the app.
4. Work the Review Queue, retrain from it, and compare each version's report before publishing.

The Review Queue is on Essentials and Business. See pricing and how AMT works.