Part numbers from a photo: 100 look-alike parts, measured

A field technician, a service desk or a returns dock has a part in hand and needs its part number. The photo is easy to take; finding the number in a catalog of look-alikes is the slow part. We trained both of StayCharted AI Model Trainer's picture models on phone photos of 100 look-alike parts, 20 photos each, and asked them about 1,000 photos they had never seen.

• The AI Image Classifier Model named the exact part on 87.8% of the 1,000 photos, out of 100 part numbers. Its own validation, measured during training, said 88.7%.
• It was nearly always right when it was sure:
  • on the 65% of photos where its confidence was 0.3 or more: 98.2% right;
  • on the 32% where it was 0.5 or more: 99.7% right (322 of 323).
• It trained in under a minute, with no GPU.
• The Dedicated AI Image Model did worse here: 75.3%. With 100 parts and 20 photos each, our reading is that a model trained on the photos themselves has too few examples per part, while the ready-made vision AI behind the AI Image Classifier Model already tells fine shapes apart. That's the opposite of our car damage benchmark, where Dedicated won.
• The mistakes are the look-alikes a person would mix up: a 1×4 plate for a 2×8 brick, a 2×3 brick for a 1×3.

For a large part catalog: start with the AI Image Classifier Model, choose the review cutoff from your own validation results, and let a person pick the part when the model isn't sure.

Why LEGO parts. They're a public stand-in for a real parts catalog, with the same hard parts: hundreds of numbers, many that differ only in length, width or one stud, and colour that says nothing about the part (every part number comes in many colours here). And each photo is labelled with the official part number.

How the test set was chosen:

• Split by source photo. The dataset cuts each brick out of a phone photo, and one photo gives several cut-outs, sometimes the same cut-out twice. So we grouped the cut-outs by the photo they came from, used one cut-out per photo, and kept each photo on one side only. No test photo shares a source photo with a training one.
• Near copies removed. We also dropped 2,725 photos that were near copies of one already kept for the same part.
• The 100 part numbers with the most photos were kept; then, per part, a fixed shuffle chose 20 photos to train on and 10 to test.
• All photos were resized so the longest side is at most 512 pixels, which is what AMT keeps and what both models read. Test photos had neutral file names.

• AI Image Classifier Model (Vision add-on, Essentials and Business): a ready-made vision AI turns each photo into a numerical representation (embedding), and a small classifier learns your part numbers from those representations. It trains in under a minute, with no GPU.
• Dedicated AI Image Model (Business with Vision): the AI itself is trained on your photos, on a GPU. It took 74 minutes here.

Projected after review assumes a person corrects every flagged mistake without introducing errors. Human review was not measured in this test.

48 of the 100 parts were named right on all 10 test photos by the AI Image Classifier Model. The hardest were parts with a near twin in the set: five parts were right on only 4 or 5 of 10.

By part family (AI Image Classifier Model)

• Plain bricks, plates and panels are the hardest. They differ by size alone, and a cut-out photo has nothing beside it to show scale.
• Some distinctive families (Technic Special, Technic Connectors, special plates and wedges) were 90% or better.

The mistakes

Most other mistakes were one or two photos each. 72 of the 122 mistakes crossed families, such as a plate taken for a brick: on a plain white background, a thin plate seen from above looks much like a brick.

Every answer comes with a confidence. Pick a cutoff, and everything below it goes to a person in the Review Queue.

AI Image Classifier Model:

Dedicated AI Image Model:

• With 100 look-alike parts, the AI Image Classifier Model is rarely very sure, even when it's right. At the default cutoff of 0.5 it sends two thirds of photos to a person, though they hold 99.2% of its mistakes. In this test, a cutoff of 0.3 reduced the review workload: a third of the photos go to a person, with projected 98.8% accuracy after review.
• For the same amount of checking, the AI Image Classifier Model was better at every cutoff. With about a third of photos checked, its projected accuracy was 98.8% against the Dedicated model's 95.1%.

Field service and maintenance. A technician photographs a worn part and gets its part number, to order the replacement. The clear cases come back at once; the uncertain ones go to the parts desk, who pick from the photo.

Service desks and spare-parts sales. Customers send a photo of the part they need. The model suggests the part number, and a person confirms the ones it isn't sure of, then quotes.

Returns and warehouses. A returned or unlabelled part is photographed at the dock and put back under the right number.

Where to start:

• Begin with your most-requested parts: 100 to 200 of them, 20 or more photos each, including real phone photos, not only catalog shots.
• Start with the AI Image Classifier Model. It trains in under a minute. On parts it beat the Dedicated model; on car damage the Dedicated model won. Test both on your own photos before choosing.
• Pick the cutoff from your own report, not the default, and let a person decide the rest.
• Add the photos people correct. Each correction is a training example for the next version, and the parts it mixes up get more examples.

• One run per model, on one public dataset. Your parts, photos, lighting and catalog will give different numbers. Validate on your own held-out photos before relying on a model.
• LEGO parts aren't industrial parts. They're cleaner and more uniform than a worn bearing on a workbench, and have no wear, oil or labels. They're also harder in one way: many differ only by size, with nothing in the photo to show scale.
• The photos are cut-outs on a white background. A real photo shows more around the part, which may help (scale, a hand) or hurt (clutter).
• A model only knows the parts it has photos of. A part with no training photos can't be named. Matching against catalog images with one picture per part is a different kind of tool (visual search).
• Some differences can't be seen. Two parts that look identical but differ in material, thread or rating need a label, a description or a measurement as well as a photo.
• The top few candidates aren't measured here. AMT gives one answer per photo today.
• The Dedicated model's result is for this setup: 100 parts, 20 photos each. With more photos per part, it may do better.

• Run through StayCharted AI Model Trainer (amt.staycharted.com) on 24 October 2026, in a dedicated benchmark account on the Business plan with Vision, exactly as a customer would:
  1. upload a ZIP of photos, one folder per part number;
  2. review data quality;
  3. train;
  4. publish;
  5. fill a ZIP of test photos;
  6. score the filled answers against the held-out part numbers.
• AI Image Classifier Model: SigLIP 2 image embeddings plus a linear classifier.
• Dedicated AI Image Model: Qwen3.5-4B fine-tuned on the training photos, on a SageMaker GPU.
• The same AI Image Classifier Model run on our development service gave the same 87.8%.

• T. M. Boiński, Images of LEGO bricks, version 1.1, Gdańsk University of Technology, 2021 (verified 2023), DOI 10.34808/arsb-4268. CC BY. See also Boiński et al., Photos and rendered images of LEGO bricks, Scientific Data 10, 796 (2023).
• LEGO® is a trademark of the LEGO Group, which does not sponsor, authorize or endorse this benchmark.
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, Tschannen et al., 2025.

• Parts classification from photos and descriptions
• Classify images in an S3 bucket
• Compare the car photo benchmark

Can AI identify a part number from a photo?

In this test, StayCharted’s AI Image Classifier Model named the exact part on 87.8% of 1,000 held-out photos across 100 part numbers. It only learns the categories represented in its training examples.

How accurate were the most confident answers?

At confidence 0.3 or above, 639 of 651 answers were correct (98.2%). At 0.5 or above, 322 of 323 were correct (99.7%). The remaining photos would need review.

How many photos were used to train each part?

The test used 20 training photos and 10 held-out test photos per part. Real catalogs may need more varied examples, especially for look-alike or worn parts.

Was the Dedicated AI Image Model better?

No. It scored 75.3% in this setup, versus 87.8% for the AI Image Classifier Model. The car damage benchmark favored the Dedicated model; compare both on your own photos.