LLM fine-tuning for private, enterprise AI: a practical guide

A general-purpose language model knows a lot about the world and nothing about how your team decides. It doesn't know which queue a ticket goes to, which clause heading your legal team uses, or which SharePoint group gets access to a new finance folder. You can describe those rules in a prompt, but a prompt alone does not train a model on years of decisions.

Fine-tuning teaches a model from your own examples instead. This guide covers what it is, when it's worth it, why it matters for private and enterprise AI, and the steps to do it well, with the settings and results from our own runs.

Fine-tuning takes a pre-trained language model (the base model) and trains it further on a smaller set of your own examples, so its answers follow your patterns.

• Full fine-tuning updates every weight in the model. It needs a lot of GPU memory and produces a full copy of the model for every task.
• LoRA (low-rank adaptation) freezes the base model and trains small adapter matrices beside it. In our runs the adapter was 0.71% of the model's parameters, and it's a file of tens of megabytes rather than gigabytes.
• QLoRA does the same, but loads the frozen base in 4-bit precision while training, which let our 4-billion-parameter run train on a single 24 GB GPU. Memory needs depend on sequence length, batch size and architecture.

For many enterprise tasks, LoRA or QLoRA is a practical starting point: fewer trainable parameters and lower memory requirements than full fine-tuning. Compare quality on your own held-out examples. The adapter is small enough to keep one per task, or one per customer.

They solve different problems, and many teams reach for fine-tuning too early, or too late.

A simple rule: if the knowledge lives in documents, use RAG. If the knowledge lives in decisions your team has already made (a column someone fills in by hand, thousands of times), fine-tune.

What the difference looks like in numbers. We scored the same held-out rows three ways: asking a strong general assistant (Claude Sonnet 5.5, given the category names and a one-line definition of each), StayCharted’s AI Classifier Model, and a fine-tuned 4-billion-parameter Dedicated AI Model. The AI Classifier compares meaning alone with words plus meaning and retains the better-scoring approach.

On BANKING77 the assistant made 2.4 times as many mistakes as the fine-tuned model. Its misses were about conventions, not comprehension: it never once chose the dataset's "get physical card" intent, because only the team's examples say when that label applies. On the medical set, where the categories follow general knowledge, the assistant won. Fine-tuning pays where the answer is your convention. Read the full benchmark and method →

Private AI depends on where training and inference run, who can access the data, and the provider's terms. Fine-tuning does not make data private automatically. It can support a private deployment in four useful ways:

1. Your patterns become an adapter. Training learns from your examples, so you do not need to include the whole training set in each request. New inputs still go to whichever system hosts inference.
2. Smaller open models can be enough. Our fine-tuned 4B model reached 93.7% on BANKING77, close to the published fine-tuned BERT result of 93.66%. That is a comparison of reported results, not a controlled head-to-head rerun.
3. You can choose the hosting environment. An open base model plus your adapter can run in an environment you control, subject to the base model's license and hardware requirements. Hosted prompting can also have contractual data protections; check the deployment and terms.
4. You can record the training and evaluation history. Keep the training dataset version, base model version, settings and held-out results. Prompt-based systems can also be logged and evaluated; neither approach creates an audit trail without those records.

The fastest return on fine-tuning in an enterprise is rarely a chatbot. It's the routine decisions that already fill your systems:

• IT service desk: category, assignment group and priority for each new ticket.
• Access management: which SharePoint or file-server groups should get access to a new site or folder, learned from existing permissions.
• Finance: GL account and cost center for each invoice line.
• Legal and records: clause types, document types and retention classes.
• Sales and support: which reply or team a new inbound message goes to.

These share three traits that make them ideal for fine-tuning: thousands of past examples already exist, the answers are your own conventions, and a person can tell whether an answer is right. They also share one requirement: the model has to say how sure it is, so unsure answers go to a person instead of straight into a system.

These are the steps we follow, with the settings from our BANKING77 run as a starting point. Treat the numbers as defaults to test, not as rules.

1. Define one decision

Write down the exact question the model answers, and the list of possible answers. "Which queue does this ticket go to?" with 40 named queues is a good task. "Help with tickets" is not. One model per decision keeps evaluation honest.

2. Collect examples of past decisions

Export rows where your team already made the decision: the ticket text and its final queue, the invoice line and its account code. A few hundred rows can work; aim for at least 20 per category, and more for categories that look alike. Use the final answer, after any corrections, not the first one.

3. Clean the data before anything else

Most fine-tuning problems are data problems. Check for:

• Duplicates, which inflate your accuracy when they land in both training and test.
• Conflicting labels: the same text with two different answers. Decide which is right, or the model learns your inconsistency.
• Near-identical categories ("Waivers" vs "No Waivers"). Merge them or write down the difference.
• Personal information the model doesn't need to decide. Remove it before training.

4. Hold back a test set

Set aside 10–20% of the rows that the model never trains on, and score it only on those. If your data comes with its own test file, use it rather than re-splitting the training file; a different split makes the score harder to compare with published results. Use separate validation rows to choose settings, and keep the final test set untouched.

5. Choose a base model

Consider size, licence, languages and where you need to run it. For classification, a modern 3–8 billion parameter open model is usually the sweet spot: large enough to read meaning, small enough to train on one GPU and serve cheaply. Pin the exact version (the repository commit, not just the name), because a bare model name can point to new weights tomorrow.

6. Choose LoRA or QLoRA

Start with QLoRA unless you have a reason not to. Our settings:

LoRA:    r = 16, alpha = 32, dropout = 0.05
Epochs:  3
Batch:   4 per device × 4 gradient accumulation steps → effective 16
LR:      2e-4
Base:    loaded in 4-bit while training

7. Format each example as a prompt and an answer

Turn each row into an instruction ("Classify this support message into one of these categories") plus the input text, with the category name as the answer. Keep the template identical between training and use. For a fixed list of categories, constrain the output at prediction time so the model can only answer with a valid category name.

8. Check the adapter covers the whole model

Make sure the adapter attaches to every layer it should. Some newer models mix two kinds of attention layer with different internal names; an adapter that targets only one kind trains happily on a fraction of the model and reports a plausible but worse accuracy. Count the adapted layers and stop the run if any are missing. (Ours: "adapted 32 of 32".)

9. Train

Our 4B QLoRA configuration fit on one 24 GB GPU. The BANKING77 run trained on 8,500 rows in 2 hours 16 minutes on one NVIDIA L4, about 16 minutes per 1,000 rows at three epochs. Expect fixed start-up costs (downloading the base, loading it) to dominate small runs.

10. Evaluate more than accuracy

On the held-out rows, measure:

• Accuracy, overall and per category. An average hides the categories the model can't do.
• Which categories it confuses. These are often ambiguities in your own taxonomy, not model failures.
• Invalid answers: anything that isn't one of your categories. Ours: 0 of 3,080.
• Confidence: whether the model's low-confidence answers are actually the wrong ones.

11. Choose a review cutoff

Pick the confidence below which a person checks the answer. This is where fine-tuning earns its keep: on BANKING77, reviewing the least-confident 9.2% of rows projected an increase from 93.7% to 98.0%, assuming every reviewed answer is corrected. This is a projection, not measured human-review performance. Choose the cutoff on validation rows, by what a mistake costs and how much checking your team can absorb.

12. Deploy, monitor and retrain

Serve the adapter on the exact base version it was trained on. Record which model and version produced every answer. Feed the corrections people make back into the training file, and retrain on a schedule. A fine-tuned model is a snapshot of your decisions; it should keep up with them.

The dangerous fine-tuning bugs don't crash. They produce a reasonable-looking number that's wrong. Watch for these:

• An adapter on the wrong base. Load a LoRA adapter on a base with the same size but different weights and it runs, answering confidently and wrongly. Check the base version before every load.
• A floating base name. "The same model" next month may be different weights. Pin it.
• Test rows in training. Duplicates and re-splits leak answers into the test set and inflate accuracy.
• A partial adapter (step 8).
• A review cutoff that flags almost nothing. In one of our early runs the confidence signal was excellent, but the cutoff was set so low that 184 of 195 errors went out marked as confident. Measure what the cutoff catches, not just what it flags.

Every step above is real work: data cleaning, GPU capacity, adapter checks, evaluation, serving and monitoring. StayCharted AI Model Trainer does them for you, from a spreadsheet, with no code.

• Upload your past decisions. A CSV or Excel file of examples with the answer your team gave each one, or a ZIP of sorted photos. No answers yet? Paid plans can group your rows and suggest starting categories to name.
• Data checks before training. It flags duplicates, conflicting labels, near-identical categories and personal information, and you fix them in a click (steps 2–3).
• A held-out score you can trust. Every model is tested on rows it never trained on, with accuracy per category and the categories it confuses (steps 4 and 10).
• The Dedicated AI Model. A language model fine-tuned on your examples with QLoRA on a GPU, with the base version pinned and the adapter checked on every run (steps 5–9). It's on the Business plan. For short, everyday text, the AI Classifier Model (which compares meaning alone with words plus meaning and keeps the better-scoring approach) often gets close, at a fraction of the training time; the report shows you which is enough.
• Confidence and a Review Queue. On Essentials and Business, answers below your cutoff enter the Review Queue. Corrections become examples for the next retraining (steps 11–12).
• Use it where the work happens. Fill files and Google Sheets, call the API from ServiceNow, Jira or your own code, connect it through Zapier, or ask it from Claude and ChatGPT.
• Private by design. Your examples train only your workspace's models. StayCharted doesn't send training data to an outside AI provider. If you connect an assistant, its provider receives what you ask it to read within your permissions. Eligible trained adapters can be downloaded on a plan that includes the model; running them separately requires the matching base model, its license and suitable hardware.

Start free → · See how it works · Compare plans

Our measurements are independent of the dataset and model authors. These are individual runs, not a ranking across all possible configurations.

• LoRA — Hu et al., 2021
• QLoRA — Dettmers et al., 2023
• BANKING77 — Casanueva et al., 2020
• LEDGAR — Tuggener et al., 2020
• LexGLUE — Chalkidis et al., 2022

Base model in these runs: Qwen3.5-4B, from the Qwen team.

Banking results and method · Contract clause results · Medical Abstracts results · Assistant comparison