Register-Consistent Biomedical Pre-Training

Language models pre-trained on biomedical papers learn medical facts in the voice of a journal article. When the same facts arrive in a patient's words ("my chest gets tight when I climb stairs"), these models do noticeably worse. BioBART is the clearest documented case: it was trained on PubMed abstracts and then lost ground on informal patient–doctor consultations such as iCliniq (BioBART). The usual explanation is a register problem. Register means the stylistic variety of language: formal scientific prose, clinical shorthand, or colloquial conversation. During pre-training, the facts and the register are learned together. The model's knowledge that a drug treats a condition is partly stored as "this word follows that word in academic phrasing."

This note proposes a small, testable change to continual pre-training. Each formal biomedical sentence is paired with a fact-preserving colloquial rewrite. The model is trained on both versions. On top of that, a loss forces its predictions for the medical content words (drugs, diseases, findings, dosages) to agree across the two versions, while the predictions for every other word are left free to follow the register. The claim is narrow. Seeing colloquial paraphrases is not enough to make knowledge register-invariant. Tying the content-word predictions across registers is what closes the gap, and it does so without the fluency loss that heavier disentanglement methods pay.

Core Claim

Treating facts as "content" and register as "style" turns the BioBART gap into a style–content disentanglement problem. Our view is that the right place to enforce the separation in a generative domain model is the model's predictions on content slots, not a split of the hidden representation. Two observations from prior work motivate this choice.

First, sentence-level text disentanglement methods use Gaussian style anchors (Multi-type Disentanglement without Adversarial Training) or a variational upper bound on the mutual information between the representation and the style label (Learning to Disentangle Textual Representations and Attributes via Mutual Information). These work on sentiment and tense in reviews. However, the second paper reports that pushing mutual information down costs content preservation and fluency. That is exactly what a domain model cannot afford.

Second, the legal pre-training work of Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining shows that separating variable factual slots from static boilerplate at the token level helps an encoder. It raises CUAD dynamic-part F1 from 0.7107 to 0.7507. That paper supplies our design principle: separate at token granularity, guided by which tokens carry facts. It does not supply a generative model or a register axis, and we add both.

The objective for a pair of formal and colloquial versions $x_f, x_c$ that share a set of content spans $E$ is

$$ \mathcal{L} = \mathcal{L}_{\text{LM}}(x_f) + \mathcal{L}_{\text{LM}}(x_c) + \lambda \sum_{e \in E} \tfrac{1}{2}\Big[\mathrm{KL}\big(p_\theta(e \mid x_f^{\setminus e}) \,\|\, p_\theta(e \mid x_c^{\setminus e})\big) + \mathrm{KL}\big(p_\theta(e \mid x_c^{\setminus e}) \,\|\, p_\theta(e \mid x_f^{\setminus e})\big)\Big]. $$

The symbols are defined as follows. $\mathcal{L}_{\text{LM}}$ is the standard text-infilling loss used by BioBART. $p_\theta(e \mid x^{\setminus e})$ is the model's distribution over the tokens of content span $e$ when $e$ is masked out of sentence $x$. $\lambda$ sets the strength of the tie. The consistency term never touches function words or register markers. This is the intended contrast with mutual-information minimization: style remains fully expressible, and only the factual prediction is required to be register-independent.

Research Program

Paired corpus construction. We will tag biomedical entities in a fixed, token-matched subset of PubMed abstracts. An instruction-tuned language model will rewrite each sentence in two target registers: patient-voice colloquial and terse clinical note. A rewrite is kept only if its tagged entity set matches the source. This filter limits fact drift. We will also audit a random sample by hand and report the hallucination rate.

Models. We will continue pre-training BART-Base and BART-Large under arms that match token count and compute exactly (see Baselines). This keeps us comparable to BioBART's architecture and its finding that sentence permutation should be dropped.

Diagnostic test set. We will build a register-gap probe: biomedical cloze and short-QA items, each in formal, clinical, and colloquial phrasing, with the facts held fixed. The headline diagnostic is the accuracy difference between the formal and colloquial versions. It measures directly what BioBART's iCliniq drop measures only indirectly.

The HTR study On the Generalization of Handwritten Text Recognition Models found that autoregressive models over-trust training-domain language statistics. Its models reach 51.8% out-of-distribution character error rate versus 40.4% for more modular CTC models. We take that result as motivation that this entanglement is real in sequence models, not as evidence about text models specifically.

Baselines

Floor references. The naive reference is general-domain BART-Large fine-tuned directly on each downstream task. Its MeQSum Rouge-L is 51.05, as reported in BioBART. The second reference is BioBART-style continual pre-training on our PubMed subset alone. Released BioBART-Large reports MeQSum Rouge-L 53.15 and a +4.45 BLEU gain on CovidDialog. We found no reported result on Lacuna for BioBART on iCliniq or HealthCareMagic, so we will measure those numbers ourselves for every arm.

Strongest competitor: paired-data augmentation. The strongest competitor is the identical pipeline trained on the same formal-plus-rewritten paired corpus with only the two language-model losses, with $\lambda = 0$. It sees exactly the same tokens, the same colloquial text, and the same compute. If the consistency term matters, this is where it has to show. A second strong competitor replaces the rewrites with an equal token budget of real unlabeled consumer-health text. This text must be drawn from sources disjoint from iCliniq and HealthCareMagic, and the arm tests whether natural colloquial data simply does the job.

Ablations. Two ablations isolate the key ingredient:

  1. Applying the consistency term to all tokens rather than content spans only. We expect this to hurt fluency in the colloquial register.
  2. Replacing it with the variational mutual-information penalty of Learning to Disentangle Textual Representations and Attributes via Mutual Information on a pooled representation against the register label.

An adversarial register discriminator is run as a further reference, though prior work predicts instability.

Fairness and the bar to beat. Every arm gets the same token count, optimizer schedule, and fine-tuning hyperparameter search budget. The best setting for each arm is chosen on validation splits. Beating the field means two things against the $\lambda = 0$ augmentation arm. First, a smaller register gap on the probe, with confidence intervals over three seeds that do not overlap. Second, a gain on colloquial downstream tasks (iCliniq, HealthCareMagic) with no loss greater than seed noise on formal tasks (MeQSum, MEDIQA).

Why not just…?

Why not just add the rewritten or real colloquial text to continual pre-training? This is the real threat, and the experiment is built around it. Its supporting evidence is that BioBART's gap is plausibly a coverage gap. Efficient Continual Pre-training for Building Domain Specific Large Language Models also shows that which domain text is chosen matters a great deal: 10% of curated data beat 100% uncurated. That suggests data choice alone could close the gap. The augmentation-only and real-colloquial arms test this directly. If either matches our method on the register-gap probe, we will report that the gap is a data-coverage problem and that disentanglement is unnecessary here. We would treat that as a useful negative result.

Why not just fine-tune on the colloquial target task? Fine-tuning adapts task format but gives little foundational knowledge, according to A Comparative Study of PEFT Methods for Python Code Generation, where pre-training mattered more than the fine-tuning method. More concretely, every arm here is fine-tuned on the target task anyway. If fine-tuning sufficed, all arms would converge on iCliniq, and the probe would show equal gaps. Both outcomes are measured.

Why not just penalize register information in the representation, adversarially or via mutual information? The adversarial version is known to collapse beyond binary labels: the loss explodes at $\lambda \ge 0.01$ on five-class FYelp in Learning to Disentangle Textual Representations and Attributes via Mutual Information. We have three registers. The mutual-information version is stable, but by that paper's own account it trades away fluency and content. We include it as an ablation. Our prediction is that it narrows the probe gap but lowers generation quality on colloquial dialogue tasks. If it does not, the token-level design loses its main justification.

Why not just pre-train register-blind syntax first, as in SAMBAL? Learning syntax without semantics shows that tiny models trained on grammatical nonsense adapt quickly to PubMed. It removes content to learn form, which is the reverse of our aim of keeping content fixed while form varies. It also says nothing about register transfer. We leave combining the two to future work rather than claim it here.

Decisive Checks

The central test is the register-gap probe at matched compute. If our method shows a smaller formal-versus-colloquial gap than the augmentation-only arm, the consistency term is doing something data alone cannot. If not, the claim is false. Two checks follow. First, colloquial-task improvement on iCliniq and HealthCareMagic must hold without formal-task regression on MeQSum, where BioBART-Large's 53.15 Rouge-L is the reference. Second, a dose-response sweep over $\lambda$ should show the gap shrinking monotonically while colloquial perplexity of non-content tokens stays flat. That combination would confirm that style was left free and only facts were tied. Finally, a leakage control will recompute all results after removing any pre-training sentence with high n-gram overlap with the test sets.