BLOG · AMAZON BEDROCK · FOR AWS PARTNERS
Why Bedrock classification pilots stall, and what gets them to production
Move an Amazon Bedrock classification pilot into production: clean labels, held-out evaluation, confidence, human review and integration. A checklist for AWS partners.
The project around the model
A customer asks for AI that sorts their work: support tickets to the right team, invoice lines to the right GL account, new products to the right category. It sounds like the easiest generative AI project there is. The customer already has the answers, in years of past decisions.
So the pilot goes quickly. Someone writes a prompt, or fine-tunes Amazon Nova on a few thousand examples, and the demo looks good. Then it stalls. Six months later it's still a pilot.
Starting a fine-tuning job in Amazon Bedrock takes minutes (here's how). The project is everything around it. These are the six places we see classification pilots stop, and what gets each one moving.
1. The data: the labels disagree with each other
Past decisions were made by different people, over years, under categories that changed. The same kind of ticket is "Billing" in one year and "Account" the next. Three categories mean almost the same thing. One has five examples.
A model trained on that learns the disagreement. Its mistakes look random, and nobody can say whether the model or the data is wrong.
What gets it moving: check the examples before training, not after. Find the same text under two categories, near-duplicates, categories too small to learn, and personal data that shouldn't be in a training file. Fix or merge, then train. A short labelling guide, written once, settles the arguments for every later version.
2. The training file: the plumbing takes the week
JSONL in Bedrock's conversation format, the same system prompt on every line and again at answer time, a validation split, an S3 bucket, a service role, KMS permissions. One malformed line fails the whole job. None of it is hard; all of it takes time, and it has to be done again for every retrain.
What gets it moving: make the file from the spreadsheet the customer already has, automatically, with the prompt fixed in one place. Delete it from the bucket when training ends, so no copy of the training data lingers in the account.
3. "How good is it?": no number anyone trusts
The demo was the ten examples that worked. The customer asks how often it's right, and the honest answer is "we haven't measured it."
What gets it moving: hold back a slice of the customer's own examples that the model never trains on, and test on them. Report accuracy per category, not only overall: a model that's 95% right overall can be 40% right on the category the customer cares most about. Show the mistakes, too. Customers trust a list of what it got wrong more than a percentage.
4. What to trust: Bedrock returns an answer, not a confidence
A fine-tuned Nova model replies with a category. It doesn't say how sure it is. Without that, every answer has to be checked (so nothing is saved) or none is (so nobody signs off).
What gets it moving: give every answer a confidence, and choose a cutoff with the customer from the held-out test. "At this cutoff, this share of answers goes straight through, this share of those is right, and the rest go to a person." A sentence like that, with the customer's own numbers in it, is what gets a pilot approved.
As an example of how much the cutoff matters: on 6,788 public CBP customs rulings, our model's exact 10-digit HTS code was right 22% of the time overall. Its surest answers, 6.7% of the total, were right 96% of the time (the benchmark). The same model is useless or valuable depending on what you do with the confidence. This CBP result is from StayCharted’s AI Classifier Model, not a Bedrock-tuned Nova model.
5. Day two: nobody owns the retrain
Categories change. New products arrive. Accuracy drifts down quietly, and the partner who built the pilot has moved on to the next project.
What gets it moving: send the unsure answers to a person, in one queue, whatever channel they came from. Every answer that person confirms becomes a training example, and the reviewer can start the next version from the same screen. The model gets better because the work gets done, not because someone schedules a project.
6. Getting answers to where the work happens
A model nobody can reach from their usual tools doesn't get used. A pilot with a notebook and an endpoint ARN isn't a product.
What gets it moving: put the answers where people already work, such as a spreadsheet they upload, a Zap or Power Automate flow, an API call from the ticketing system, a Google Sheet, or their AI assistant.
Why a model trained on their examples, not a better prompt
A prompt can describe ten categories. It can't describe the hundred small rules a team applies without writing them down. On tasks that follow a company's own rules, models trained on past decisions make fewer mistakes: on banking messages, a model trained on about 10,000 labelled examples made 6.3 mistakes in 100 against 15.3 for a leading general AI, and on contract clauses 15.1 against 23.1 (how we measured). Where the task follows general knowledge, the two come out close, and a prompt may be enough. Test both on the customer's own data and let the numbers decide.
How StayCharted does the six steps, in your customer's AWS account
StayCharted AI Model Trainer is the product around the fine-tuning job:
- The data: checks the examples before training and reports what it found.
- The file: writes the JSONL, starts Bedrock's fine-tuning of Amazon Nova Micro, Lite or Pro in your customer's own AWS account, and deletes the file when training ends. Connecting is one CloudFormation template; no access keys.
- The number: tests on held-out examples and reports accuracy per category, with the mistakes.
- The confidence: gives every answer one, with a cutoff the customer chooses.
- Day two: unsure answers go to the Review Queue; confirmed ones train the next version.
- The channels: spreadsheet fills, the API, Zapier, Power Automate, Google Sheets, and Claude or ChatGPT through our connector.
The tuned model stays in your customer's account and answers on demand on their AWS bill. StayCharted is on AWS Marketplace.
For AWS partners: your customer can subscribe to StayCharted’s public AWS Marketplace listing. Earn through your services, resell directly through StayCharted’s reseller program, or build on the Partner API. StayCharted for AWS partners →
Common questions about Bedrock classification pilots
Why do generative AI classification pilots fail to reach production?
Rarely because of the model. Usually it's inconsistent training labels, no measured accuracy on held-out data, no confidence to decide what a person checks, and no owner for retraining.
How do I evaluate a fine-tuned Bedrock model for classification?
Hold back examples the model never trains on, ask it with the same prompt it was trained with, and compare its answers with your team's, per category. Keep the list of mistakes.
Does Amazon Bedrock give a confidence score for a fine-tuned model's answer?
Not for a tuned Nova model's text answer. You need to estimate one, for example from how similar past examples were labelled, and check the estimate against held-out data.
How many examples do I need to fine-tune for classification?
A few hundred is a start for a handful of clear categories. Many categories, or fine distinctions between them, need more. The held-out test tells you whether you have enough.
Should I fine-tune or just prompt?
Fine-tune when there are hundreds of examples and categories that follow the company's own rules. Prompt when categories are few and follow general knowledge. Test both.