When Does an LLM Need Specialist Training?

Fine-tuning is often reached for as the default answer to a model that isn't good enough. It is also one of the more expensive answers and the cheaper fixes are frequently overlooked. In a recent Passion Academy, ML engineer George Marmaras looked at when an LLM actually needs specialist training. He drew on two case studies, SaulLM in law and Strata in medicine and finished with what a first adaptation experiment might look like.

‍

Slides

‍

‍

‍

The goal is measurable improvement on a defined task, not a "better" model in the abstract.

‍

Start With the Task, then Diagnose the Gap

‍

In law, that might mean reviewing contract clauses against a rubric, flagging risks/ citing the supporting text with fewer missed risks and fewer false alarms. In medicine, it might mean estimating the probability of a future clinical event from the patient history available so far, with more reliable estimates as the goal.

‍

Once the task is defined, the next question is where the model falls short:

‍

Missing evidence: The relevant source information never reaches the model, whether that is a clause or a patient record. The fix is to supply the evidence in the prompt or add retrieval (RAG). No amount of training helps a model reason about a document it never sees.

‍

Persistent task errors. The evidence is already in front of the model and it still gets things wrong. Here the order is to check prompts and examples first, then test specialist training.

‍

The two aren't mutually exclusive, since retrieval and training can work together. The point is to diagnose the gap before choosing the intervention.

‍

‍

Three Ways a Model Learns

‍

If training is warranted, there are three main learning signals.

‍

1. Continued pre-training learns from domain text. The model keeps predicting the next toke, but on documents from the target domain, which builds familiarity with terminology and drafting patterns.
2. Supervised fine-tuning (SFT) learns from worked examples, such as an input paired with a target answer. It also uses next-token prediction but the aim is better task execution rather than broader familiarity.
3. Reinforcement learning learns from rewards. The model generates answers, a reward function scores them (by rules or a learned evaluator), and the model is updated to favour higher-scoring outputs. The catch is that the reward has to reflect the behaviour we actually want.

‍

‍

Reminder of how a Base Model Learns

‍

A base model learns from large text collections by continuing text ("The capital of France is" → "Paris."), while an instruct model has been further trained on examples of desired responses so that it follows requests and formats. OpenAI's route from GPT-3 to InstructGPT in 2022 combined all three: pretraining for broad language capability, supervised fine-tuning on human-written answers to improve instruction following, then reinforcement learning on rewards learned from human-ranked answers. Continued pretraining is a separate step. It means taking an existing model and pretraining it further, often on domain data.

‍

‍

Case Study: SaulLM and Broad Legal Specialisation

‍

SaulLM-7B takes a general base model, Mistral-7B and adapts it for law in two stages.

‍

In the first stage, legal continued pre-training, the model trains on roughly 30 billion tokens of cleaned and de-duplicated English legal text. This covers legislation, court decisions, contracts and corporate filings from the US, Canada, the UK, Europe and Australia. The purpose is familiarity with legal terminology, argumentation and document structure.

‍

In the second stage, legal instruction tuning, the team built training examples in three steps:

‍

• Turning legal text and known labels (such as document type) into initial question-and-answer pairs
• Using Mistral-7B-Instruct to generate follow-up questions and answers
• Combining these synthetic conversations with general instruction data.

‍

The result is SaulLM-7B-Instruct, which follows user requests, answers questions and summarises legal material.

‍

The results on LegalBench-Instruct (average balanced accuracy) show what each stage contributes:

‍

‍

The base model improves dramatically from continued pretraining alone, before it has seen any dedicated instruction tuning. Also legal-specific instruction examples add a further two percentage points over generic ones. The research question was whether continued pre-training on a large legal corpus can improve the legal capabilities of an existing general-purpose LLM. On the benchmarks studied, yes. Follow-up work introduced SaulLM-54B and SaulLM-141B, and the 141B model surpassed the study's GPT-4 baseline on LegalBench-Instruct.

‍

SaulLM is an example of broad domain specialisation and it needed a very large corpus to get there.

‍

‍

Adapting Efficiently: LoRA and QLoRA

‍

Not every project has 30 billion tokens or the budget to update every weight. This is where parameter-efficient methods come in.

‍

Full fine-tuning updates all model weights and every version of the adapted model has to be stored in full. LoRA (Low-Rank Adaptation) instead keeps the original weights frozen and represents the update to each weight matrix using two smaller trainable matrices. The rank controls the adapter's capacity and parameter count. As an illustration, a single 1,024 × 1,024 matrix has over a million parameters to update in full (1,048,576), while a rank-8 LoRA trains just 16,384. Each version becomes a small adapter rather than a whole model.

‍

The natural question is whether you give up quality for that saving. The Thinking Machines Lab piece "LoRA Without Regret" suggests that, under suitable conditions, you don't. With sufficient rank, LoRA can closely track full fine-tuning, while low-rank adapters fall behind once capacity becomes the limiting factor. Layer selection and training settings also affect the comparison.

‍

QLoRA goes a step further by storing the frozen base weights in 4-bit form and back-propagating through the quantised base into the adapters. LoRA reduces the number of trainable weights and QLoRA also reduces base-model storage.

‍

‍

Case Study: Strata and Narrow Clinical Extraction

‍

If SaulLM is the broad approach, Strata is the narrow one. Strata is an open-source framework for developing, fine-tuning, evaluating and deploying local models for clinical information extraction. It was evaluated on breast and kidney pathology, prostate MRI and bone-marrow reports.

‍

The problem is a practical one. Research databases need consistent structured fields across many reports but the information lives in narrative text. A pathology report saying the specimen is a left breast core biopsy with a final diagnosis of invasive ductal carcinoma has to become a record with tissue source, cancer present and cancer subtype. It is a narrow, repeated task, which gives us a clear, measurable target for adaptation.

‍

Strata starts from Llama 3.1 8B Instruct and applies supervised fine-tuning with LoRA. It does this using clinical reports paired with human-labelled fields, split into separate training, development and test sets. The pre-trained weights stay frozen and there is one adapter per dataset.

‍

On average exact-match accuracy across the four clinical extraction datasets, where every requested field in a report must be correct, the results were:

‍

‍

The amount of labelled data needed was also modest. The training sets contained 100 (breast), 95 (kidney), 43 (prostate) and 72 (bone marrow) reports and each dataset-level model trained for one to three hours on a single A40 GPU. That was enough to match the study's second annotator on each dataset.

‍

"Human level" means comparison with a second trained annotator, on selected extraction tasks from two institutions. New report styles and new tasks still require local evaluation.

‍

‍

Designing a First Experiment

‍

George closed by sketching what a first specialisation experiment could look like for a project turning customer documents into validated structured records.

‍

1. Define the output: Agree on the fields, annotation rules and the errors that matter.
2. Establish the baseline: Test an existing model with a well-developed prompt on held-out documents.
3. Test adaptation: Train LoRA adapters on reviewed examples, using QLoRA if memory is limiting.
4. Compare the outcome: Measure exact-match quality, serious errors, runtime and total cost.

‍

The experiment succeeds if adaptation improves a useful customer outcome, not just a metric.

‍

‍

The Broader Picture

‍

The two case studies sit at opposite ends of the spectrum. SaulLM shows what broad specialisation looks like: a large domain corpus, two training stages and a model with wider legal capability.

‍

Strata shows what narrow specialisation looks like: a defined extraction task, a few dozen to a hundred labelled reports, a lightweight adapter and a few hours of GPU time.

‍

The second is far closer to what most real projects can afford to try.

‍

Across both, the answer to "when does an LLM need specialist training?" comes down to three conditions:

‍

1. A persistent gap. Errors remain after improving prompts and access to relevant evidence.
2. Useful training data. Examples or documents address the capability the task requires.
3. Trustworthy evaluation. Held-out cases show whether the improvement generalises.

‍

Without the first, training is premature. Without the second, it has nothing to learn from. Without the third, you can't tell whether it worked. Specialist training is a powerful tool but it earns its place only when a measured task gap remains and the data and evaluation are there to close it and prove it.

‍

‍

References

• COLOMBO ET AL, SAULLM-7B: A PIONEERING LARGE LANGUAGE MODEL FOR LAW, ARXIV:2403.03883
• LIU AND LIAN, HUMAN LEVEL INFORMATION EXTRACTION FROM CLINICAL REPORTS WITH STRATA
• THINKING MACHINES LAB, LORA WITHOUT REGRET

‍