STAYCHARTED BLOG · MACHINE LEARNING · SEARCH RELEVANCE
Machine learning for search relevance: when a model trained on your judgments wins
A model trained on your past relevance judgments scored 80.2% on WANDS against 72.7% for a general AI. On ESCI they were level. Here is why, and when to use it.
What search relevance classification is, and where machine learning fits
Search relevance classification takes a pair (a search and a product) and gives it a grade. The two datasets we tested use grades most teams will recognize:
- Exact match: the product is what was searched for, with every detail asked.
- Partial match (WANDS) or Substitute (ESCI): the right kind of product, with something different, such as the size, style or brand.
- Complement (ESCI only): not what was searched for, but used with it, such as a case for a phone.
- Irrelevant: not what was searched for.
Teams use these labels to build evaluation sets for a ranking change, to watch search quality week to week, and to find queries where the results have gone wrong. The work is slow because each grade needs a person to read the query and the product together. A query product relevance model does the first pass on every pair and sends the doubtful ones to a person.
There are three ways to automate it. Rules and keyword overlap are cheap but miss meaning (a "sofa" result for "couch"). A general AI language model, given the label names and definitions, reads meaning well but applies its own idea of what "Partial match" means. A model trained on your team's past judgments (supervised machine learning) learns your team's idea of it.
How accurate machine learning for search relevance is: the benchmark
We tested on 4,000 held-out query-product pairs per dataset, after training on 10,000 pairs. Each search went wholly to training or wholly to test, so the models were graded on searches they had never seen. The general AI was Claude Sonnet 5.5 through Amazon Bedrock, given the answer names and one-line definitions but no examples. Full method: Search relevance benchmark.
| Dataset | Labels | Always guess the most common label | StayCharted AI Classifier Model | StayCharted Dedicated AI Model | Claude Sonnet 5.5 |
|---|---|---|---|---|---|
| WANDS (Wayfair, furniture and home) | 3 | 60.5% | 55.6% | 80.2% | 72.7% |
| Amazon ESCI (US English) | 4 | 44.5% | 44.2% | 59.0% | 59.9% |
Three things stand out:
- WANDS: the Dedicated AI Model, an AI language model fine-tuned on the 10,000 past judgments, led by 7.5 points.
- ESCI: Claude led by 0.9 points. That is one run on one dataset, so we read it as a tie, and we would not tell you a trained model wins here.
- The AI Classifier Model scored below simply guessing the most common label on both sets. For relevance pairs we do not recommend it. Use the Dedicated AI Model or a general AI instead.
Why a model trained on your relevance judgments can beat a general AI
Every relevance team has a standard that the label definitions only partly capture: how different the size has to be before an Exact match becomes a Partial match, or whether a matching sofa cover counts as Irrelevant to "sofa". Your judges settle those cases the same way hundreds of times, and the answers live in your past judgments, not in a one-line definition.
WANDS shows this clearly. Wayfair's reviewers called most results "Partial match." The trained model learned that habit. A general AI given only definitions applied a more generic standard and split those pairs more often into Exact or Irrelevant.
| WANDS label | Test rows | StayCharted Dedicated AI Model | Claude Sonnet 5.5 |
|---|---|---|---|
| Partial match | 2,419 | 91.8% | 68.2% |
| Irrelevant | 1,183 | 65.8% | 83.8% |
| Exact match | 398 | 52.3% | 67.6% |
The trained model won overall because it matched the standard the judges applied to the most common case. It was worse on the two smaller labels. Averaged equally across labels, Claude scored 73.2% against 70.0%. If your team cares most about catching Irrelevant results, that matters, and the fix on the trained side is more examples of the smaller labels plus a confidence cutoff that sends doubtful answers to a person.
Why ESCI came out level
ESCI is hard for every model. Its four labels are fine distinctions, and some of its labels look inconsistent: the page notes a belt with a buckle marked "Exact match" for a search for belts without a buckle. A trained model can only learn a standard that the judgments apply consistently. When the judgments disagree with each other, a model trained on them inherits the noise.
The per-label results point the same way. The trained model was better on Exact match (64.8% against 53.9%), the largest label. Claude was better on Substitute, Irrelevant and especially Complement (66.5% against 39.8%, on only 161 test rows). Averaged equally across labels, Claude led 61.9% to 52.1%.
The honest reading: training on your own judgments pays off when those judgments follow a steady, team-specific standard. If your labels are noisy, clean them up first, or expect a general AI to do about as well.
LLM vs trained model for relevance: consistency and confidence
Accuracy is one number. Two others decide how much of the work a team can hand over.
Consistency. We re-ran 500 rows. Claude changed its answer on 2.8% of WANDS rows and 7.6% of ESCI rows. A published model version applies the learned categories to new pairs; repeat-run consistency was not measured for the trained model here. For a relevance evaluation set that you compare week to week, a label that moves between runs makes a ranking change look better or worse than it is.
Confidence you can act on. The trained model returns a confidence from 0 to 1 with every answer, so you can choose a cutoff, accept the answers above it and send the rest to review. The benchmark projected the effect at each cutoff:
| Cutoff | WANDS: share sent to review | WANDS: projected accuracy | ESCI: share sent to review | ESCI: projected accuracy |
|---|---|---|---|---|
| 0.70 | 10.3% | 86.1% | 20.8% | 71.3% |
| 0.80 | 16.5% | 87.8% | 31.3% | 76.4% |
| 0.90 | 28.6% | 90.1% | 45.7% | 83.5% |
| 0.95 | 40.8% | 92.4% | 58.8% | 89.1% |
These are projections, not measurements. They assume every wrong answer sent to review gets corrected. They still show the trade you can make: on WANDS, reviewing about 29% of pairs projects to about 90% accuracy overall. There is more on picking a cutoff in AI confidence scores and human review.
Volume. Claude used about 1.1 million input tokens and 340,000 output tokens on the 4,000 WANDS pairs, and about 1.5 million and 611,000 on ESCI. A general AI bills on every pair you send. A trained model in AMT runs on a fixed monthly plan, which is worth weighing if you judge relevance on large volumes every week.
Test your search relevance model on searches it has never seen
One result in the benchmark matters for anyone building relevance judgments with AI. On WANDS, the AI Classifier Model scored 82.9% when the test rows were picked at random, and 55.6% when the test used searches it had never seen. With a random split, the same search appears in both training and test, so the model partly remembers answers instead of learning the standard.
Your production traffic will include new searches every day. Hold out whole searches when you test, and treat any score from a random-row split with suspicion.
When you don't need machine learning for search relevance
Training is not always worth it. A general AI with good definitions is a fine choice when:
- You have a few broad labels and the line between them is obvious.
- You have few past judgments, or judgments that disagree with each other often (as ESCI suggests).
- Your rarer labels, such as Irrelevant or Complement, matter more than overall accuracy. In this benchmark the general AI did better on them.
- The stakes are low and a person looks at every result anyway.
The benchmark also did not test few-shot prompts, retrieved examples or prompt tuning for Claude, any of which could narrow the gap. If you already run your own language model, AMT's "Your AI Model" option gives it your categories and similar past rows from your data. It is not trained, but it is a fair middle ground.
How to start with ecommerce search relevance labeling in AMT
StayCharted AI Model Trainer (AMT) is built for this loop:
- Export past judgments. One row per pair: the query, the product text (title, brand, product type) and the label your judges gave.
- Upload and check. AMT checks the data, including personal-information checks and masking before training.
- Train a Dedicated AI Model. It is available on the Business plan (pricing). In the benchmark it took about three hours for 10,000 rows.
- Read the test report. AMT tests on held-out rows the model never saw and shows accuracy overall and per label, the most common confusions, and how many answers above each confidence cutoff are right.
- Fill new pairs. Use a CSV or Excel file, the Google Sheets add-on, the API, or a Snowflake or Databricks table. Each answer comes with a confidence and a needs-review flag.
- Review and retrain. Answers below your cutoff go to the Review Queue. Your judges' corrections train the next version.
Your data stays in your organization's own database area, is used only for your models and is hosted on AWS in the US.
Common questions
What is machine learning for search relevance?
It is training a model on your team's past relevance judgments (query, product and grade) so it can grade new query-product pairs the same way. In AMT each answer comes with a confidence, so doubtful ones go to a person.
How accurate is a search relevance classification model?
In our benchmark on 4,000 held-out pairs, a Dedicated AI Model trained on 10,000 past judgments scored 80.2% on WANDS (3 labels) and 59.0% on Amazon ESCI (4 labels). Your accuracy depends on how consistent your own judgments are.
Is an LLM better than a trained model for relevance judgments?
It depends on your data. On WANDS, the trained model beat Claude Sonnet 5.5 by 7.5 points. On ESCI, Claude was 0.9 points ahead, which is effectively level. Claude did better on the smaller labels in both sets. The trained model gave the same answer every time; Claude changed 2.8% to 7.6% of answers on a re-run.
How many relevance judgments do I need to train a query product relevance model?
The benchmark trained on 10,000 pairs per dataset. You can start with fewer and let the test report tell you whether accuracy is good enough. Make sure each label, especially the rare ones, has enough examples.
Can AI replace human relevance judges?
It can take the first pass. With a confidence cutoff, answers the model is sure of go straight through and the rest go to the Review Queue. Your judges' corrections then train the next version, so they keep setting the standard.
What is the difference between search relevance classification and query classification?
Relevance classification grades a query against a specific product (Exact, Partial, Irrelevant). Query classification sorts the query alone into your catalog categories. See site search query classification.