Machine Learning for Contract Clause Classification: What 100 Clause Types Show

Clause classification combines three things that make automation difficult.

• Many categories. LEDGAR has 100 clause types. Random guessing would score about 1%, and the rarest types appear only a handful of times.
• Long, specialized text. A typical provision runs 87 words, about one in ten exceeds 230 words, and the longest is nearly 1,000 words.
• Labels that sit right next to each other. "Waivers" and "No Waivers", "Definitions" and "Defined Terms", "Applicable Laws" and "Governing Laws", "Modifications" and "Amendments". The difference between these is a convention, and conventions are rarely written down in full.

Your own taxonomy adds a fourth problem. A legal or procurement team's clause library reflects years of decisions about where a borderline clause belongs. Those decisions live in the labels, not in a style guide.

Keyword rules

Rules such as "contains 'governing law' → Governing Laws" are quick to write and easy to explain. They break down when two categories share vocabulary, and with 100 types they become a large rulebook that someone has to maintain.

Prompting a general LLM with your category names

You give an assistant the list of categories, perhaps with short definitions, and ask it to pick one per clause. This needs no labeled data. It does rely on the category names carrying all of your team's conventions, which is hardest exactly where labels overlap.

A legal clause classification model trained on your examples

You give the model thousands of clauses your team has already labeled. It learns the boundaries from those decisions, is tested on clauses it never saw, and then returns a category and a confidence for each new clause. This is supervised machine learning, and it is what StayCharted AI Model Trainer (AMT) does.

All figures below come from the same 4,000 held-out test clauses. The trained models learned from 10,000 labeled clauses. Claude received the category names and one-line definitions with no labeled examples. Full method: legal clause classification on LEDGAR and asking an assistant vs. training a classifier.

What the table shows:

• The Dedicated AI Model led by 8.0 points over Claude. It made 15.1 mistakes per 100 clauses against Claude's 23.1, about a third fewer.
• The close labels are where prompting struggled. Claude labeled 34 "Waivers" clauses as "No Waivers" and 34 "Definitions" clauses as "Defined Terms". The fine-tuned model made those two confusions 10 and 11 times.
• Cheaper trained models were close behind. Words and meaning reached 82.7%, and plain word matching with logistic regression reached 82.3%. The Dedicated AI Model's lead over them was 2.2 and 2.6 points.
• These are single runs. The assistant comparison reports approximate 95% margins of ±0.9 to ±1.8 points across its tests, so a 2-point gap is suggestive rather than a firm ranking. The 8-point gap over Claude is well outside that range.

The fine-tuned model still confused related labels. Of its 603 mistakes, 114 fell among its fifteen most frequent confusions. Cleaning up overlapping labels before training is one of the most useful things a legal team can do.

It learns your conventions from examples. A category name like "No Waivers" tells an assistant very little about which clauses your team files there instead of "Waivers". Thousands of labeled clauses show it directly.

It reads long clauses as they are. The Dedicated AI Model reads up to 4,096 word-pieces of a clause when labeling, so a 1,000-word provision is handled in full.

Use a published model version. Claude returned the same category on 98.8% of legal clauses when the identical request was repeated on the first 500 rows, so about 1 in 100 answers changed on a second ask. In the full run, 138 replies had to be retried because they were empty or cut off, and two named no valid category at all. The model is trained to return your categories; keep its version recorded with each result.

Its confidence is tested against known answers. AMT measures, on held-out clauses, how many answers fall above each confidence cutoff and how many of those are right. As the assistant benchmark notes, a confidence value stated by an assistant is not automatically calibrated to your task.

More examples help steadily on contract text. These are the words and meaning results (the AI Classifier Model approach) at each training size:

Claude's 76.9% with no examples sits between the 20 and 50 per type results. The honest reading: if you only have a few labeled clauses per category, prompting a general LLM is competitive. Training pulls ahead once you have dozens of examples per type, which most clause libraries and CLM exports already contain. More on sizing in how many examples you need to train an AI classifier.

A model that is 84.9% accurate still gets about 15 in 100 clauses wrong, so the useful question is which ones to check. In AMT, answers below your cutoff are flagged and go to the Review Queue. Reviewers' answers then train the next version.

For the Dedicated AI Model on LEDGAR:

Two cautions apply. The after-review figures assume each flagged mistake is corrected without introducing a new error; they were not measured with human reviewers. The cutoffs were also chosen after seeing test results, so set yours on a separate set of clauses with known answers.

Contract text needs more checking than simpler tasks. On banking messages, the same kind of model flagged 9.2% at a 0.90 cutoff and projected about 98% after review. On contract clauses, reaching a similar level meant reviewing roughly 42% to 54% of clauses, depending on the approach. Plan for review as part of the workflow. See AI confidence scores and human review.

• A few broad categories. If you only separate a handful of clearly different clause families, rules or a prompted assistant may be enough.
• A small one-off batch. If a lawyer will read every clause anyway, a model adds little.
• No labeled history yet. With fewer than about 10 examples per type, the table above shows prompting holds up. Label a sample first, then train.
• Tasks where the assistant already does well. On a medical abstracts benchmark, Claude scored 65.7% against 63.6% for the Dedicated AI Model. Training does not win everywhere, so test on your own data.

1. Export clauses with their approved labels. A clause library, a CLM migration export or a procurement contract repository works well. AMT classifies clause text you supply, so split contracts into clauses first.
2. Settle overlapping labels. Have legal decide how pairs like "Waivers" and "No Waivers" should be used before training.
3. Upload to AMT. It checks the data, runs personal-information checks and masking, and trains a private model. Your data stays in your organization's own database area, is used only for your models, and is hosted on AWS in the US.
4. Pick a model. The AI Classifier Model is on every plan (Free includes one). The Dedicated AI Model is on Business. Compare plans on the pricing page.
5. Read the test report. Check accuracy per category, the main confusions, and the share of answers above each cutoff.
6. Fill new clauses. Use CSV, Excel or JSON files, the Google Sheets add-on, the API (Essentials and up), or Snowflake and Databricks tables.

For the legal workflow in more detail, see contract clause classification for legal teams and document classification. For the full product, see how AMT works.

The Review Queue requires Essentials or Business. Retrain and publish a new version to use reviewer corrections in future answers.