Machine learning for medical text classification: when a general LLM wins, and when training still helps

The assistant vs trained model benchmark ran the same held-out test rows through three approaches on three public datasets. Claude received the category names and one-line definitions, with no labeled examples. The trained models learned from about 10,000 labeled rows per dataset.

On banking and legal text, training won by 9.0 and 8.0 points. On medical abstracts the Dedicated AI Model was 2.1 points behind Claude, or 36.4 errors per 100 rows against Claude's 34.3. The page gives approximate 95% margins of ±1.7 points for Claude and ±1.8 for the Dedicated AI Model, and notes that a paired analysis has not been done, so the gap is small. We still report it as a loss for the trained models, because that is what the numbers show.

The life sciences medical abstract benchmark describes the data in detail: 14,438 published abstracts, five condition groups (Neoplasms, Digestive system diseases, Nervous system diseases, Cardiovascular diseases and General pathological conditions), and a fixed test set of 2,888. Random guessing scores 20%, and always picking the largest group scores 33.3%. Three features of the task explain the result.

Five broad categories that follow general medical knowledge

A trained model earns its keep by learning distinctions that live only in your team's history. These five groups are the opposite. The comparison page puts it this way: they "draw on general medical knowledge." A large language model has read a great deal of medical writing, so a category name such as "Neoplasms" already means something to it. With only five groups, there are few fine boundaries for training data to teach.

Overlapping disease groups

General pathological conditions is the largest group (33.3% of the test set), and the abstract benchmark notes that "many abstracts about a general condition also concern a specific organ system." That overlap is where most errors sit: 821 of the fine-tuned model's 1,051 mistakes (78%) involve this group. The most common single confusion was an abstract labeled general pathological conditions that the model called cardiovascular diseases, 171 times. Claude struggled with the same group, scoring 40.2% on it.

When the labels themselves overlap, more training examples cannot fully separate them, and no approach on the page did well on that group.

Confidence did not let much work go straight through

On a well-separated task, a trained model's confidence lets most rows go straight through. On these five overlapping groups, it did not. For the fine-tuned model, the abstract benchmark shows:

Compare the banking benchmark, "where a 0.90 cutoff flagged 9.2% of messages for a projected 98%". The projections assume a reviewer fixes every flagged error, and human review was not measured. They show that on this task a person would still read most abstracts.

The public test is five fixed groups. Real literature triage work rarely looks like that. A trained medical text classification model is worth comparing when one or more of these is true:

• Your own categories. Inclusion and exclusion decisions for a review, safety-relevant versus not, your therapeutic area tags or your internal routing rules are conventions your team built. The comparison page says "Categories shaped by your team's conventions are a reason to compare training on your own examples." On BANKING77 (77 intents) and LEDGAR (100 clause types), where categories are many and specific, training won by 8 to 9 points.
• Volume. The comparison page notes that an AI Classifier Model "runs on CPU without sending a prompt and examples with each prediction." When you sort many thousands of abstracts, that matters. The benchmark did not measure cost per row or end-to-end speed, so we do not quote a figure.
• Consistency. Claude gave the same category on repeat requests for 97.0% of the first 500 medical rows. A trained model gives the same answer to the same text every time.
• A confidence you can check. AMT tests every model on held-out rows and shows the share of answers above each cutoff and how many of those are right, so you can see how much review a cutoff will need before you rely on it.
• Cost of training. The cheaper approach trained quickly: the embedding model behind the 60.0% result took "about 4 minutes" on CPU. The fine-tuned model took "about 8¼ hours on one NVIDIA L4." Pick the model kind to match the gain you measure.

The comparison page is direct: "For a one-off sort, or categories that follow general knowledge, it may be enough." That covers a lot of clinical text classification AI work at small scale: a few hundred abstracts sorted into broad disease areas for a one-time landscape review, for example. If your categories look like a textbook's chapter list, try prompting first.

AMT also offers "Your AI Model": your own LLM is given your categories and similar past rows from your data. It is not trained, and it can be a sensible first step while you collect labeled examples. A trained model can be added later and compared on the same held-out rows.

The honest way to choose is to test both on your data. In AMT:

1. Export abstracts or documents your team has already sorted, with the category each received. The abstract benchmark suggests: "For a first model on overlapping categories, a hundred good examples per group goes most of the way." See how many examples you need.
2. Write down the tie-break rules. The abstract benchmark advises: "If an abstract can belong to both "general" and a specific system, write down which wins."
3. Upload to StayCharted AI Model Trainer. AMT runs personal-information checks before training. It detects patterns such as email addresses, phone numbers and US Social Security numbers, and lets you mask them. It does not detect names, street addresses or health details, so remove those yourself if they are in the text.
4. Train an AI Classifier Model (every plan, Free includes one). On Business, also train a Dedicated AI Model. AMT reports accuracy overall and per category, confusions and the confidence table.
5. Pick a cutoff. Answers below it go to the Review Queue (Essentials and Business), and reviewers' answers train the next version.
6. Fill new rows from CSV or Excel files, the Google Sheets add-on, the API, or Snowflake and Databricks tables.

For more on cutoffs, read AI confidence scores and human review and human-in-the-loop machine learning. For other document types, see document classification. Plans are on the pricing page and the feature list is on the product page.

• It sorts literature and documents into categories. It does not diagnose, recommend treatment or classify patients.
• The benchmark used published research text with no patient records. Nothing in it is medical advice, and AMT does not give medical advice.
• Results were single runs on English abstracts and five broad groups, with cutoffs chosen after seeing test results. Your categories may score differently, which is why AMT tests on your own held-out rows.