Image and text classification together: classify photos with the notes that come with them

A photo shows what something looks like. The note says what the photo cannot: where it is, when it started, what the person heard or was told, what was already tried. Your team's past decisions were made with both in front of them, and a model trained on the photo alone learns only half of that.

Some common pairs:

These pairs are examples of where text and photos meet in real work. They are not benchmark results.

You build a training file in the usual way, with one row per decision. Each row has a photo, the text column you want the model to read, and the category your team gave it.

• A photo and a note. The picture model reads both and learns from both. The train an image classifier without code guide puts it this way: "a picture model can read a text column alongside each photo and learn from both."
• A note but no photo. The row is still answered from its text, and the answer is marked "no photo". That keeps a fill from failing when a field report or claim arrives without an attachment, and it tells your reviewers which answers were made from words alone.
• Answers and review. Every answer comes with a confidence from 0 to 1 and a needs-review flag below your cutoff. Unsure answers go to the Review Queue, and reviewers' answers train the next version when a Builder or Reviewer retrains.

Both picture model kinds can read the text column. The AI Image Classifier Model needs the Vision add-on on Essentials or Business. The Dedicated AI Image Model, which fine-tunes the image model on your examples, needs Business with the Vision add-on.

No benchmark measures photo plus text together. We have not tested how much a note adds to a photo, or how a photo-and-note model compares with a photo-only model or a text-only model, so we do not quote a figure for this combination.

What we have published are photo-only benchmarks. They show what the picture models do with images alone, and nothing more:

• Car damage, photos only. On the automotive damage benchmark, with 6 categories (front or rear, by breakage, crushed or not damaged) and 552 test photos, the AI Image Classifier Model scored 76.6% and the Dedicated AI Image Model 86.4%. On 8 damage types, the scores were 64.1% and 70.2%.
• Part identification, photos only. On the part identification benchmark, with 100 look-alike part numbers, 20 training photos each and 1,000 test photos, the AI Image Classifier Model scored 87.8% and the Dedicated AI Image Model 75.3%.

No text was used in either test. When you train a photo-and-note model in AMT, its held-out test is the measure that counts: AMT reports accuracy overall and per category, confusions, and the share of answers above each confidence cutoff with how many of those are right.

Add the text column when:

• The deciding fact is in the words. If two photos look alike but your team gave them different categories because of what the note said, a photo-only model cannot learn that rule.
• Photos are sometimes missing. Rows with no photo are answered from text and marked "no photo", so one model covers the whole queue.
• Your team already writes consistent notes. Short, specific notes in your own vocabulary are what a model can learn from.

Leave it out when:

• The photo decides. If the category is plain from the picture (a grade of fruit, a type of product), the note may add little. Train photo-only first.
• The text is pasted boilerplate. A note that says the same thing on every row teaches nothing.
• The text is the whole decision. If the photo rarely matters, a text model is simpler. See document classification.

Because we have not measured the combination, the practical approach is to train both ways on the same rows and compare the held-out results in AMT before you rely on either.

Insurance and warranty claims

A claim usually arrives with photos and a description. A model trained on your past triage decisions can suggest a route or a damage category from both, send sure answers straight through and send the rest to an adjuster. See insurance claims triage. The model does not settle a claim or decide coverage. It gives your team a suggestion and a confidence.

Visual inspection and condition monitoring

Inspectors often photograph a defect and type a short note. Training on both lets the model learn the grade your inspectors would give. See visual inspection and condition-based monitoring from inspection photos, which describes a spreadsheet with a link to each photo, its grade and the technician's note.

Parts in the field

A technician's photo of a part, with a typed description, is a natural photo plus description pair. See identify parts from a photo and the mobile app.

1. Export past decisions with a link to each photo, the note or description, and the category your team gave it. You need at least 10 photos per category, and 20 or more is better.
2. Clean up category names first. "Water damage" and "water-damaged" count as separate categories until you merge them.
3. Check the text for personal details such as names and addresses before uploading, especially in customer complaints.
4. Upload to StayCharted AI Model Trainer, set up the picture model with your photo references, accompanying text and category labels, and train an AI Image Classifier Model.
5. Read the held-out test report and pick a cutoff. Answers below it go to the Review Queue.
6. Fill new rows, and retrain with your reviewers' answers.

Plans are on the pricing page and the full feature list is on the product page. For how cutoffs and review work together, read AI confidence scores and human review.