SOLUTIONS · FOR AI PRODUCTS
Triage feedback on your AI product into your own issue types, automatically
Sort thumbs-down feedback on your AI product into your own issue types and owners, with a model trained on feedback your team already sorted.
AI products often collect feedback: a thumbs-down, a "this is wrong", a one-line complaint. At a few a day, someone reads them all. At thousands a week, they pile up unread, and the pattern that would tell you what to fix is in the pile. StayCharted sorts each one into your own issue types, using a model trained on the feedback your team has already sorted, so your connected workflow can route it to the team that owns it.
Why your own issue types
A general AI can tell "negative" from "positive". Without your rules and examples, it does not automatically know your product’s failure modes or who owns them:
- "It used last year's price list" is a stale data problem for the data team.
- "It booked the wrong room" is a tool-call problem for the agent team.
- "Way too long" is a prompt or format issue for product.
Your team has already made hundreds of these calls while reading feedback. Those decisions are the training data.
What it reads, and what it answers
It reads the feedback comment, and optionally the question and the AI's answer it was about:
| Field | Example |
|---|---|
| Comment | "It quoted the old refund policy" |
| User's question (optional) | "Can I return an opened item?" |
| AI's answer (optional, shortened) | "Yes, within 14 days…" |
It answers with your issue types, and the owning team if you train for that too:
- Wrong or outdated information → Data
- Wrong tool or action → Agent team
- Misunderstood the question → Prompting
- Format or length → Product
- Not about the AI (billing, account) → Support
How to set it up
1. Export feedback your team already sorted
From your feedback tool, issue tracker or spreadsheet: one row per piece of feedback, with the issue type your team gave it. A few hundred reviewed rows can be a starting point; coverage of each issue type matters more than the total.
2. Train
Upload the file and train an AI Classifier Model. It reads meaning and many languages. To learn both issue type and owner in one answer, use an answer with several parts.
3. Sort the new feedback, every day or as it arrives
- As it arrives: a Zapier or Make step, or your backend through the API, sends each new piece of feedback to the model and creates an issue in the owning team's queue.
- In batches: export the week's feedback to a sheet and fill it with the Google Sheets add-on.
4. Review the unsure ones, count the rest
API answers below your cutoff enter the Review Queue on Essentials and Business. Google Sheets fills are reviewed in the sheet instead. Use your issue tracker or spreadsheet to count categories and compare them week by week.
Put the feedback comment first and shorten long AI answers so the useful context fits the model’s input window. API workflows require Essentials or Business; the Sheets add-on is available on every plan.
What it doesn't do
- It doesn't check whether the AI was actually wrong. It sorts what users reported. Verifying the answer is your team's (or an evaluation tool's) job.
- It doesn't collect feedback. Keep your current thumbs-up/down and observability tools; this works on what they export.
- New issue types need examples before the model can use them.
Evaluate before automating
Use reviewed examples that reflect your current rules. Check errors by category and sample confident answers as well as uncertain ones. Confidence is a model score, not a guarantee. Keep mandatory safety and access rules in your application. Best practices for training and review →.
Common questions
How do you categorize user feedback on an LLM app?
Train a model on feedback your team already sorted into issue types. It sorts new feedback the same way, with a confidence, and sends the unsure ones to a person.
Can it route feedback to the right team?
Yes. Train it on the owning team as well, as a second answer or a second model, and connect it to your issue tracker through Zapier, Make or the API.
How much sorted feedback do I need?
About 20 per issue type to start, and more for types that are easy to confuse.
Does it read the AI's answer too?
It can. Add the question and the answer as extra columns, and the model uses them along with the comment.