Keeping the Prompt Slot Intact: Domain Pre-Training for Few-Shot Clinical Classifiers

Many clinical text tasks come with only a few dozen labeled examples, such as dementia detection from interview transcripts or a hospital-specific note classifier. Two common pieces of advice for this setting point in opposite directions. The first is to continue training the language model on unlabeled text from the target domain before fine-tuning. This step is called domain-adaptive pre-training (DAPT), and its purpose is to teach the model the domain's vocabulary. The second is to keep the model general and phrase the task as a fill-in-the-blank prompt. This is called cloze-style prompting: the model reads the input followed by a sentence like "The participant is [MASK]" and predicts a label word such as "healthy" or "impaired," whose reliability depends on its prediction for that blank. The evidence suggests these two steps can interfere with each other. Continued domain training can make a model worse at few-shot prompting even while it learns the domain.

This note proposes a specific explanation and a cheap fix. The explanation is that domain training shifts what the model predicts at the masked position inside task-style templates, and prompt-based few-shot learning depends on exactly that prediction. The fix adds one term to the domain-training loss. Unlabeled domain sentences are wrapped in the task template, and the adapted model's prediction at the blank is kept close to the original model's prediction. All other parts of the model are left free to learn the domain. The claim to test is that this separates what DAPT gains from what it costs. If it works, a model can absorb clinical vocabulary without losing the prompt behavior that makes learning from 8 to 64 examples possible.

What the literature establishes, and what it leaves open

Three results motivate the proposal.

The first comes from Effect of Multilingual and Domain-adapted Continual Pre-training on Few-shot Promptability. It compares a biomedical continually pre-trained Llama3-8B with the unadapted base model and finds that the adapted model often prompts worse on domain tasks such as BC5CDR named-entity recognition. Supervised fine-tuning then restores performance. The authors did not test whether the domain-training phase itself could be designed to avoid the loss.

The second comes from Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training. It shows that direct fine-tuning with generated augmentation and self-distillation (DFT++) beats continually pre-trained models at 2 to 5 examples per class, reaching 78.90% accuracy on 5-shot BANKING77. In short, DAPT's advantage disappears or reverses in the few-shot regime, at least for short dialogue utterances.

The third comes from Domain Adaptation via Prompt Learning for Alzheimer's Detection. Its method, DAPF, uses a composite cloze template on clinical speech transcripts and reaches mean F1 0.8952 on CCC→ADReSS. It does no domain pre-training at all, and the authors leave open whether adding it would help or hurt.

Two lines of work supply the methods this proposal reuses. Do not Mask Randomly masks domain keywords at a higher rate during DAPT. On PUBHEALTH it reports 65.40 F1, compared with 64.94 F1 for random masking and 63.23 F1 for no adaptation. This is the domain-training objective used here. Efficient Domain Continual pretraining by Mitigating the Stability Gap and FDAPT supply the generic ways to protect against forgetting. These are mixing general text back in and freezing early layers, and both appear below as competing methods.

Two related papers address the case where the target task is too small for ordinary DAPT. TADPOLE handles it by selecting relevant Wikipedia sentences for a short extra round of pre-training. AdaPrompt retrieves open-web passages using prompt-shaped queries. Both decide which text the model trains on. Neither controls what the model is allowed to change.

That distinction is the difference from the closest work. AdaPrompt is the nearest neighbor, because it links continued pre-training to prompt-based few-shot learning. However, it assumes domain training is harmless to prompting and optimizes data selection. This proposal assumes domain training actively damages prompting, and it targets the specific output it damages.

Core Claim

Prompt-based few-shot learning reads the model's prediction over the vocabulary at the masked position of a template. Domain training on clinical text moves that prediction toward domain keywords, away from the label words the template relies on. Generic forgetting defenses do not target this damage. Mixing in general text exercises the blank prediction only in general contexts, and freezing layers does not protect the output head's behavior in domain contexts.

The proposed objective is

$$ \mathcal{L} = \mathcal{L}_{\text{kw-MLM}}(x) + \lambda \, \mathrm{KL}\big(p_{0}(\cdot \mid T(x)) \,\|\, p_{\theta}(\cdot \mid T(x))\big). $$

The symbols are as follows. $x$ is an unlabeled in-domain passage. $\mathcal{L}_{\text{kw-MLM}}$ is the keyword-masked language modeling loss from Do not Mask Randomly. $T(x)$ wraps $x$ in a task-style cloze template, with the model's prediction read at the template's [MASK] position. $p_0$ is the frozen original model's prediction at that position, and $p_\theta$ is the prediction of the model being adapted. $\lambda$ sets the strength of the anchor.

The anchor uses no labels, so it fits a setting where only a few labeled examples exist.

The claim is falsifiable in three ways. It fails if plain DAPT does not hurt few-shot prompt-based learning in our setting, because then there is nothing to fix. It fails if the anchor does not recover that loss. It also fails if general-text mixing matched for compute recovers the loss equally well.

Research Program

The work has three stages.

The first stage measures the problem in the encoder cloze setting, which is the setting where clinical few-shot prompting actually works (DAPF). A BERT-base or RoBERTa-base model is continued on domain text with keyword masking. Checkpoints are saved throughout. At each checkpoint we record three quantities. The first is zero-shot cloze accuracy with a fixed template. The second is few-shot prompt-based fine-tuning F1 at 8, 16, 32 and 64 labeled examples. The third is the KL divergence between the checkpoint's and the base model's predictions at the template blank on held-out unlabeled text. Together these give a "promptability curve," analogous to the stability-gap curve of Guo et al.. The mechanistic prediction is that drift at the blank tracks the loss in few-shot F1 better than general-domain perplexity does.

The second stage trains with the anchored objective. It sweeps $\lambda$ and the number of templates used for anchoring. One condition is decisive for whether the method generalizes beyond the template it was trained on. In that condition, the anchor uses templates that are disjoint from the evaluation template. The test then measures whether the model preserved prompt behavior in general or only memorized one template. This matters because DAPF reports that small wording changes, such as "Patient" versus "Participant," move accuracy noticeably.

The third stage is a smaller replication on a decoder model. It asks whether the same anchor, applied to the next-token distribution after an in-context instruction, reduces the BC5CDR loss in promptability that Yano and Miwa observed. This stage is run at reduced scale and reported as supporting evidence, not as the main result.

Data and settings

The primary benchmark is clinical speech: DementiaBank/ADReSS and the Carolinas Conversation Collection, in both cross-domain directions used by DAPF. Unlabeled transcripts from the source corpus and from the training split of the target serve as the DAPT text.

Two secondary benchmarks test whether the results hold outside speech. The first is PUBHEALTH, which lets us compare directly with the keyword-masking numbers. The second is CHEMPROT and RCT20K from the TADPOLE suite, subsampled to few-shot sizes, with PubMed-style text as the DAPT corpus.

All labeled sets are subsampled to 8, 16, 32 and 64 examples per class, with five random draws each. The main metric is macro-F1. We also report the variance across random draws and the spread across templates.

Baselines

The naive reference point is prompt-based fine-tuning of the unadapted base model, which is DAPF as described in Domain Adaptation via Prompt Learning for Alzheimer's Detection. It is reproduced with the same templates and the same small labeled sets. This is the strongest existing method on the primary benchmark, and its reported figures are mean F1 0.8952 on CCC→ADReSS and F1 0.7881 on DB→CCC. Those numbers come from the paper's own data regime, which is not our 8 to 64 shot subsamples, so we rerun DAPF on our subsamples. Our method beats it only if it improves macro-F1 by more than the across-draw standard deviation at most of the four sample sizes.

The second baseline is plain DAPT followed by prompt-based fine-tuning. Domain training uses keyword masking as in Do not Mask Randomly, whose reported PUBHEALTH result is 65.40 F1 with full training data. No reported few-shot result for this method on our benchmarks was found on Lacuna. This baseline isolates the cost the anchor is supposed to remove.

The third family of baselines covers generic forgetting defenses, each applied to the same DAPT run. These are general-text mixing at the ratio style of Guo et al., freezing of earlier layers as in FFDAPT from FDAPT, and a low-rank adapter version of DAPT. They are the strongest competitors on the question asked here, because each is a simpler way to "not forget."

The fourth baseline is DFT++ from Revisit Few-shot Intent Classification. It is run without DAPT, and its augmentation comes from a local open model of similar size.

The fifth baseline is a TADPOLE-style selection round from TADPOLE, covering the case where domain text is itself scarce.

The key ablation applies the KL anchor to unwrapped text, meaning arbitrary masked positions in raw passages instead of the template blank. If this works equally well, the benefit is general regularization and not protection of the prompt slot.

Fairness rules apply to all comparisons. Every DAPT variant gets the same token budget and optimizer steps. Every method gets the same hyperparameter search budget, about 20 trials, chosen on a held-out labeled split of the same size as the training split. All methods are evaluated on identical subsample draws.

Why not just…?

Why not just fine-tune directly with augmentation and skip DAPT? This is a real threat. DFT++ shows that at 2 to 5 examples per class, DAPT models lose to direct fine-tuning. That result, however, comes from short dialogue utterances with little vocabulary shift. Clinical transcripts and biomedical abstracts carry heavy terminology shift, the condition Naik et al. identify as covariate shift. Generated augmentation also risks invented clinical content. The DFT++ arm, compared across sample sizes, settles where the crossover lies. If DFT++ wins at every sample size on the clinical benchmarks, the honest conclusion is that domain training is not worth its cost here, and we would report that.

Why not just mix general text back in or freeze layers during DAPT? These methods protect general ability, and they may well protect prompting too. The hypothesis is that they do not, because the damaged behavior is the blank prediction in domain contexts, which general text never exercises. The matched-compute comparison against mixing, freezing and adapters decides this directly. If any of them recovers few-shot F1 as well as the anchor, the anchor offers nothing beyond them.

Why not just rely on fine-tuning to recover, as Yano and Miwa found? Yano and Miwa observed recovery under supervised fine-tuning with ordinary amounts of data. Whether fine-tuning on 8 to 64 examples has enough signal to repair a shifted blank prediction is exactly what is unknown. The plain-DAPT baseline measured across sample sizes answers this. The expectation is that the gap closes at 64 examples but not at 8. If the gap is already closed at 8, the problem the anchor targets does not arise in this regime.

Decisive Checks

The proposal stands or falls on four results.

First, plain keyword-masked DAPT must lower 8- and 16-shot macro-F1 relative to the unadapted DAPF baseline on at least the ADReSS/CCC transfer directions. Without this loss there is nothing to fix.

Second, drift at the template blank must correlate with that loss across checkpoints. Domain perplexity must correlate with it more weakly.

Third, the anchored objective must recover the loss at small sample sizes while matching or exceeding plain DAPT at 64 examples. It must also beat compute-matched general-text mixing, freezing and adapters. The anchor's own ablations must hold up as well: anchoring on unwrapped text should be weaker, and anchoring on templates disjoint from the evaluation template should still help.

Fourth, the preservation must not be confined to the training template. The spread of F1 across evaluation templates should not grow relative to the base model.

Any one of these failing narrows the claim. The first or third failing removes it.