Search relevance, measured: a model trained on past judgments against a general AI

Does this product page answer this search? Stores, marketplaces and SEO teams judge that by hand every day. We took two public sets of real shopping searches with people's judgments, trained a model on 10,000 past judgments, and asked it, and Claude, about 4,000 query–product pairs held out from StayCharted’s training, using search queries absent from its training split.

• Wayfair (WANDS), furniture and home:
  • The trained model was right on 80.2% of the 4,000 test pairs. Claude was right on 72.7%.
  • With a person checking the tenth of answers the model was least sure about, it would reach a projected 86.1%, assuming every flagged error is corrected.
• Amazon (ESCI), everything a store sells:
  • The two were level: 59.0% against 59.9%, a small descriptive difference in this run.
  • This set is hard for everyone. Its four answers are fine distinctions, and some labels appear inconsistent.
• The trained model needed no instructions: only the past judgments. Claude was given a one-line definition of each answer.
• The trained model gave the same answer every time. Asked the same 500 rows twice, Claude changed its answer on 2.8% (Wayfair) and 7.6% (Amazon) of them.
• The test itself can mislead. Tested on random rows of its own file, one model scored 82.9%. On searches it had never seen, the same model scored 55.6%. Section 5 explains why, and what we changed.

Every test search is new. A search usually has many judged products. If “red velvet sofa” appeared in both training and test, performance could benefit from exposure to that query rather than measure generalization to new searches. So each search went wholly to training or wholly to test, by a fixed hash of its id, and no test search appears in training.

The answers mean:

• Exact match: the product is what was searched for, with every detail asked.
• Partial match (WANDS) or Substitute (ESCI): the right kind of product, with something different, such as the size, style or brand.
• Complement (ESCI): not what was searched for, but used with it, such as a case for a phone.
• Irrelevant: not what was searched for.

• Dedicated AI Model (StayCharted, Business plan): a language model trained on the 10,000 past judgments, on a GPU, through AMT exactly as a customer would use it: upload, train, publish, fill a file. It was given nothing else: no definitions, no instructions.
• AI Classifier Model (StayCharted): a pre-trained AI represents the meaning of each row and a small classifier learns the answers. This benchmark used meaning embeddings plus a linear classifier on CPU.
• Claude Sonnet 5.5, through Amazon Bedrock: given the answer names, a one-line definition of each and the row, and asked for one answer per row. This is how a team would ask a general AI to do the job.

Claude wasn't given example judgments in its instructions. Few-shot prompting, retrieved examples and prompt optimization were not tested. These can be automated and could change the results.

The observed WANDS gap is 7.5 percentage points in favor of StayCharted’s Dedicated AI Model. On ESCI, Claude leads by 0.9 points. These are descriptive differences from one run. Rows share queries, so a simple independent-row margin of error does not capture all uncertainty; we did not calculate a query-clustered significance test.

Where each one is right

Two things to read from these:

• The trained model matched the common WANDS label more often. On WANDS, Wayfair's reviewers called most results "Partial match", and the trained model recovered that label more often in this test. Claude, from definitions alone, calls more of them Irrelevant or Exact.
• Claude found more of the rare answers in this test. Claude found more of the less common answers (Exact match on WANDS; Complement and Irrelevant on ESCI). The training data included only 406 Complements in 10,000 rows; this comparison does not isolate why the models differed. Averaged over the answers equally, Claude is ahead on both sets (WANDS 73.2% against 70.0%; ESCI 61.9% against 52.1%).

The AI Classifier Model struggled on these relevance pairs

It scored at or below always giving the most common answer. The tested configuration represented each pair with meaning embeddings. Relevance depends on the relationship between the query and product; these results suggest that this configuration did not capture it well enough. It suits single-text jobs: which page group a search belongs to, or which category a product is in. See AI keyword mapping with Search Console.

Same row, same answer. Asked the same 500 rows a second time, Claude gave a different answer on 14 (2.8%, WANDS) and 38 (7.6%, ESCI). A trained model gives the same answer to the same row every time, so a file filled twice is filled the same way.

A confidence on every answer. The trained model says how sure it is, so people can prioritize unsure answers for review; confident mistakes can still pass through:

Projected accuracy assumes every incorrect answer selected for review is corrected. It is not a measured human-review result.

On WANDS, checking 10.3% of answers would raise accuracy from 80.2% to a projected 86.1% under that assumption. On ESCI, reaching the same point means checking about half, which says this task, as Amazon labeled it, still needs people for much of the work.

What Claude used. For each set of 4,000 rows, Claude read about 1.1 million (WANDS) and 1.5 million (ESCI) tokens and wrote 340,000 and 611,000, most of it reasoning before the answer. The definitions are sent with every row. A trained model reads only the row.

Before you trust any model's accuracy, check how it was tested.

AMT used to test every model on a random 15% of the rows in its file. That's fine when every row stands on its own. In these files, each search has many rows, one per judged product, so a random test puts most test searches in training too: the model has seen other rows for nearly every search it's tested on.

On WANDS, the AI Classifier Model tested that way scored 82.9%. On searches it had never seen, it scored 55.6%. The first measures performance on a random-row split with overlapping queries; the second measures generalization to held-out queries. Choose the evaluation that matches your deployment.

The same thing happens wherever rows belong together:

• many tickets from the same customer, or emails in the same thread;
• many leads from the same company;
• many products from the same supplier;
• many rows for the same search query.

What we changed. AMT now asks how a model should be tested, on the same screen where you choose its columns:

• Random rows, for files where every row stands on its own;
• Keep rows that share a value together: choose a column, such as customer, search or thread, and every row with the same value is either tested or trained on, never both;
• Test on the newest rows: choose a date column, and the model is tested on its newest rows of each category after learning from the older ones.

When a column's name says rows belong together, AMT suggests keeping them together, and the validation report says which test was used.

• If you have past judgments, train on them. Relevance reviews, merchandising decisions and site-search fixes are years of the judgment you want repeated. On WANDS a model trained on 10,000 of them beat a general AI by 7.5 points, with nothing else written down.
• Where the line is general knowledge, a general AI keeps up. On ESCI's four fine distinctions, Claude matched the trained model from one-line definitions. Know which kind of task you have before you choose.
• Plan for review. No engine fully automates relevance judgments. Use the confidence to send the unsure answers to a person, and train again on what they correct.
• Test on searches the model hasn't seen. Keep each search's rows together when you test, or the number will promise more than you'll get.
• For mapping searches to pages, a single-text job, the AI Classifier Model is the inexpensive choice. See AI keyword mapping with Search Console.

• One run of each model, on two public datasets. Your searches, products and reviewers will give different numbers. Test on your own held-out searches first.
• The labels disagree with themselves. In ESCI, for example, a belt with a buckle is marked "Exact match" for a search for belts without a buckle. Agreement with dataset labels is not independently verified correctness; reviewer agreement was not measured here.
• The product pages are short: a title and a brand (ESCI), or a name and a type (WANDS), not the full page.
• Claude was given definitions, not examples. Section 2 explains why. With chosen examples in every request, its numbers may differ.
• Training time isn't compared. The trained model took about three hours on one GPU for 10,000 rows in our development environment; the AI Classifier Model's run waited on a small shared server there. Neither says much about production speed.

• Data: 10,000 training and 4,000 test rows from each dataset. Searches were split by a fixed hash of their id, so no test search is in training, and rows were drawn with their answers in their natural proportions. ESCI used the US English "small" version. Each row read: "Search query: … / Page: …".
• Dedicated AI Model: Qwen3.5-4B, trained with LoRA on the training rows, on an AWS GPU (ml.g6.xlarge), through StayCharted AMT's development environment, 15 October
  2026. Answers are chosen from the labels, with a confidence from the model's own probabilities.
• AI Classifier Model: multilingual-e5-small meaning, plus a linear classifier, the same environment and date.
• Claude: us.anthropic.claude-sonnet-5-5 on Amazon Bedrock, 15 October 2026, with the answer names and one-line definitions, one row per request. Replies that ran out of room while reasoning were asked again with more room, and the answer the reply ended on was taken. No reply was left without an answer.
• Scoring: each engine's answers on the same 4,000 test rows, compared with the datasets' own labels. Scores are descriptive; no query-clustered significance test was calculated.

These public datasets may have appeared in pre-training material for either underlying model. We did not audit that exposure. Query-disjoint here refers to StayCharted’s supervised training split, not a guarantee about pre-training data.

• WANDS: Chen, Y., Liu, S., Liu, Z., Sun, W., Baltrunas, L. & Schroeder, B. (2022). WANDS: Dataset for Product Search Relevance Assessment. Proceedings of the 44th European Conference on Information Retrieval (ECIR 2022). Wayfair, MIT license.

• ESCI: Reddy, C. K., Màrquez, L., Valero, F., Rao, N., Zaragoza, H., Bandyopadhyay, S., Biswas, A., Xing, A. & Subbian, K. (2022). Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. arXiv:2206.06588. Amazon, Apache 2.0 license.

• Wayfair WANDS dataset, paper citation and MIT license

• Amazon ESCI dataset and Apache 2.0 license

• Shopping Queries Dataset paper

These sources describe the datasets. StayCharted’s benchmark runs supplied the model results reported above.

Is a fine-tuned model better than an LLM for search relevance?

On 4,000 held-out query–product pairs per dataset, StayCharted’s Dedicated AI Model scored 80.2% on WANDS versus Claude Sonnet 5.5 at 72.7%. On ESCI, scores were close: 59.0% versus 59.9%. Results depend on the dataset, labels and method; few-shot prompting was not tested.

Can AI judge search relevance automatically?

AI can help prioritize relevance judgments, but mistakes remain. Reviewing the least-confident 10.3% of WANDS answers would raise 80.2% accuracy to a projected 86.1% if every flagged error were corrected. This is a projection, not a measured reviewer result, and confident errors can remain.

Why not give Claude examples instead of training?

Giving Claude examples is a valid alternative that this benchmark did not test. Examples can be selected manually or retrieved automatically. Training offers a maintained model learned from labeled history; compare both approaches on your own held-out queries.

What is train-test leakage?

When the same thing, such as a customer or a search query, appears in both the rows a model trains on and the rows it's tested on. The test may benefit from related examples and overstate performance on genuinely new groups. Keep related rows together when you test.

Which datasets were used?

WANDS (Wayfair, MIT license) and the Shopping Queries Dataset, ESCI (Amazon, Apache 2.0 license), US English.