Same Reasoning, Lay Words: Concept-Weighted Self-Distillation for Medical Reasoners

Language models trained heavily on medical literature reason well when a question is written the way a textbook would write it. Patients do not write that way. "My chest gets tight when I climb stairs and it goes away when I sit" and "exertional chest pain relieved by rest" describe the same finding, but a model may handle the second correctly and the first poorly. The two phrasings share almost no words. This is the vocabulary mismatch problem, long studied in retrieval, now showing up inside multi-step reasoning. We propose to measure how much accuracy specialized medical models lose when a question is moved from technical to lay wording, and to close that gap with one training signal. The model reads the technical version of a question and produces a reasoning trace. It is then trained so that, reading the lay version, it reproduces that trace. The training loss is weighted so that the clinical concepts, negations and quantities in the trace count much more than the connective prose around them.

The core claim fits in two sentences. A medical model that reasons correctly from technical wording already contains the knowledge it needs, so the lay-wording failure is a failure to map lay descriptions onto the concepts it reasons with, and that mapping can be taught by self-distillation from the model's own technical-wording behavior. Because most tokens in a reasoning trace are formatting and connective filler, uniform distillation spends its signal on the wrong tokens, and weighting by semantic importance should be what makes the transfer work.

Why this gap, and why now

Three findings in the literature point at this gap. BioBART continued pre-training BART on PubMed abstracts and found that the resulting model does well on scientific text but weakens on informal clinical dialogue such as iCliniq. That is a "clinical gap" caused by the register of the pre-training data. PMC-LLaMA shows the payoff of domain pre-training: a 13B model trained on 79B tokens of papers and textbooks reaches 64.43% averaged over PubMedQA, MedMCQA and MedQA, against 54.97% for ChatGPT. It also inherits a literature-shaped view of patients, overestimating severe pathology. Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? introduces a conditional consistency score: the probability of solving an easy variant given that the model solved the hard one. It shows 7B models sit between 50% and 70% even when the variants differ only superficially. Together these suggest that the more a model specializes in formal text, the more fragile it may become to a change in surface register. That fragility has not been measured for multi-step clinical reasoning.

A fourth finding limits what we can claim. Disentangling Reasoning and Knowledge in Medical Large Language Models shows that only 32.8% of MedQA questions actually require multi-step reasoning; the rest are recall. A register gap measured on all of MedQA would mostly measure vocabulary lookup. We therefore evaluate on the reasoning-heavy subset that paper identifies, and report recall-heavy items separately.

Core Claim

Retrieval research handles vocabulary mismatch at the front door. RaFe trains a small query rewriter with preference signals from a reranker, which turns conversational queries into index-friendly terms and gains 2 to 3 points of downstream QA accuracy. WikiQA documents that 20.3% of correct answers share no content words with the question. Both treat the gap as a single translation step before any reasoning happens.

Our claim is different. In clinical reasoning, translating lay words into technical terms is itself a diagnostic act. Rewriting "tight chest on stairs" as "angina" commits to an interpretation before any reasoning is visible. We want the reasoner itself to recover the concepts. We would show this by training it to produce, from lay input, the same concept-level trajectory it produces from technical input.

The method has three ingredients:

  1. Teacher pass. The domain-pretrained model reads the technical version $x^{tech}$ and generates a reasoning trace $y$. Only traces that reach the correct answer are kept.
  2. Student pass. The same model, with gradients on, reads the lay version $x^{lay}$ and is scored on reproducing $y$.
  3. Weighted loss. We minimize

$$ \mathcal{L} = \frac{1}{\sum_t w_t}\sum_{t} w_t \,\mathrm{KL}\big(p_\theta^{sg}(\cdot \mid x^{tech}, y_{<t}) \,\|\, p_\theta(\cdot \mid x^{lay}, y_{<t})\big), $$

Here $p_\theta$ is the model's next-token distribution and $sg$ marks a stopped gradient on the teacher pass. $t$ indexes positions in the trace, and $w_t = 1 + \alpha\, s_t$ is the importance weight. In that weight, $s_t \in \{0,1\}$ marks tokens inside a span linked to a medical concept, a negation scope, or a numeric quantity with its unit, and $\alpha$ sets how much more those tokens count.

The marker $s_t$ is built from concepts rather than parts of speech. Fixed part-of-speech weights are known to break exactly where medicine is most sensitive: negations, units and qualifying modifiers. Because technical terms fragment into several sub-words, weights are assigned to the whole linked span rather than to single tokens.

Prior work: roles

Several papers supply the weighting idea. Tiantianzhu7's STS system and syntactic tree matching for entailment weight content words over function words when comparing texts. CLEME2.0 weights edits by their effect on meaning. All three use weighting to score text after the fact. We use it as a training-loss weight, and we replace part-of-speech classes with linked concepts.

Other papers share the instinct that reasoning should be weighted by meaning but act at a different point. Semantic Self-Consistency weights whole sampled reasoning paths by embedding similarity at inference time and gains up to 13.53% on StrategyQA for Llama-2-7B. Revision Distance counts LLM-proposed revisions and explicitly leaves non-uniform weighting of those revisions open. Med-REFL values reflection steps by downstream answer correctness and reaches 73.74% on MedQA from Huatuo-o1-8B. None of these targets robustness to input register. Each operates on paths, revisions or reflection steps, whereas we weight individual positions in a cross-input distillation loss.

Domain pre-training defines the regime we work in and the limits we work within. PMC-LLaMA and BioBART are the kind of models we start from. A Comparative Study of PEFT Methods for Python Code Generation argues that domain depth of the base model predicts downstream performance more than the fine-tuning method does. That cautions us against claiming a fine-tuning-stage method injects knowledge, and it is why we frame the method as remapping existing knowledge rather than adding new knowledge.

exBERT and VE-KD fix sub-word fragmentation of domain terms by extending the vocabulary, but only in encoders. We use fragmentation as an analysis variable rather than a component, asking whether register gaps are larger for questions whose technical terms shatter into many pieces. ADAPT reweights whole samples during pre-training by similarity to an anchor set. Our weighting is finer-grained, at the token level, and conditioned on meaning rather than on similarity to an anchor set.

Research Program

Stage 1: build the paired benchmark. For the reasoning-heavy MedQA subset identified by Disentangling Reasoning and Knowledge, we produce lay rewrites of each vignette in the patient's voice. Each rewrite replaces technical findings with symptom descriptions and keeps lab values as a patient would report them. Clinicians check a stratified sample to confirm that the rewrite still determines the same answer, and items that fail are dropped. We define the register consistency score as $P(\text{lay correct} \mid \text{technical correct})$, the direct analogue of the ConsisEval score, alongside plain lay accuracy.

Stage 2: measure the gap and its correlates. We measure the gap for a general instruction model (Llama-3.1-8B-Instruct), a literature-pretrained model (PMC-LLaMA 13B), and a reasoning-tuned medical model (Huatuo-o1-8B or BioMed-R1-8B). Against the gap we relate two quantities: how many sub-words the gold concepts fragment into, and concept recall in the trace, meaning the share of gold concepts the trace names, judged with an entity linker.

Stage 3: train with weighted self-distillation. Training pairs come from two sources. Natural lay-to-expert pairs come from MeQSum consumer questions and their expert summaries. Synthetic pairs come from MedQA training vignettes, with no overlap with test items. We apply the loss above to each model.

Stage 4: check for side effects. We confirm that technical-register accuracy, robustness to adversarial wrong-diagnosis prefixes, and generation on iCliniq and HealthCareMagic do not degrade.

Baselines

The do-nothing reference is each model's zero-shot accuracy on lay and technical versions with the same chain-of-thought prompt. This sets the size of the gap. Beating it is necessary but uninteresting.

The strongest existing competitor is rewrite-then-reason. A small rewriter is trained RaFe-style (RaFe) to turn lay vignettes into technical ones, with preference signals from how well the downstream reasoner answers. The unchanged reasoner then answers the rewrite. RaFe reports 2 to 3 points of downstream QA accuracy on open-domain retrieval. We found no reported result on Lacuna for rewrite-then-reason on lay-register medical reasoning. For fairness, the rewriter gets the same training pairs and the same gradient-step budget as our method, plus an equal hyperparameter search. Beating it means higher lay accuracy on the reasoning-heavy subset with confidence intervals that do not overlap. A tie at lower inference cost also counts as a meaningful practical win, since our method needs no second model at inference.

A second strong competitor is plain fine-tuning on paraphrase-augmented data. The model is fine-tuned to produce the teacher trace given the lay input with ordinary cross-entropy, which amounts to uniform weighting with hard targets. It gets the same data and steps.

The ablations that isolate the key ingredient are:

  • uniform-weight KL, with $\alpha = 0$;
  • part-of-speech weights, with nouns and verbs up-weighted;
  • a semantics-free alternative that weights each position by its current teacher-student divergence, which up-weights wherever the student is wrong without knowing what a concept is;
  • concept weights with negation and quantity spans removed.

The proposal's central bet is that concept weighting beats both uniform and divergence weighting. If divergence weighting matches concept weighting, the semantic component is not doing work and we will say so.

Why not just…?

Why not just rewrite lay questions into technical language first? This is the real threat, and the experiment is designed to settle it. Our reason to expect it to fall short is that a rewriter must interpret symptoms to produce technical terms, so its errors are diagnostic errors made out of sight. The reasoner cannot recover from a confident wrong rewrite. We will test this by splitting the lay items by whether the rewriter's output preserves the gold concepts, and comparing both systems on items where it does not. If rewrite-then-reason matches us overall, that is a finding in itself: the register gap would be a front-end translation problem. We would then reframe the contribution as the benchmark and the diagnosis.

Why not just use a bigger general model? PMC-LLaMA's results (PMC-LLaMA) suggest domain models outperform larger general ones on technical wording, which is why practitioners deploy them. The open question is whether that advantage survives lay wording. Stage 2 measures exactly this, and we include a 70B general model as a reference. If the general model has a small gap and our method does not bring the domain model's lay accuracy above it, the method has no practical use.

Why not just weight tokens by where the student is currently wrong? Divergence weighting needs no entity linker and is a fair, simpler competitor. Our concern is that, on lay input, the largest divergences sit on stylistic tokens: the student phrases things differently, not wrongly. If so, divergence weighting re-concentrates the signal on filler. It is run as an ablation under identical budgets, and the comparison is decisive for whether semantic importance matters.

Why not just expand the vocabulary so technical terms stop fragmenting? Fragmentation affects how technical terms are represented, but lay phrasings are mostly common single tokens. Fixing technical-term fragmentation does not by itself connect "tight chest" to the angina concept. Our Stage 2 correlation tells us whether fragmentation explains much of the gap. If it does, vocabulary extension in the style of exBERT becomes a follow-up rather than an alternative.

Proposed Experiments and Decisive Checks

The first decisive check is whether the gap exists. If PMC-LLaMA and the reasoning-tuned models show a register consistency score close to their technical accuracy on the reasoning-heavy subset, the premise fails and we stop. We expect a gap larger for the literature-pretrained model than for the general instruct model, consistent with BioBART's clinical gap.

The second check is the head-to-head. On held-out lay MedQA reasoning items, concept-weighted distillation should beat uniform KL, divergence weighting, and paraphrase fine-tuning, and at least match rewrite-then-reason. Technical-register accuracy should change by no more than the run-to-run noise.

The third check is mechanism. Concept recall in the first steps of the lay-input trace should predict final correctness, and the weighted method should raise that recall more than the ablations do. A gain in accuracy without a gain in concept recall would mean the model found some other shortcut, and would undercut the explanation.

The fourth check guards against a specific failure. Up-weighting concepts could teach the model to ignore small lay qualifiers. We build a contrast set of lay items where a negation or quantity flips the answer ("the pain does not get worse when I breathe in"). Accuracy there must not fall below the uniform-KL baseline. This is also why negation and quantity spans are part of the weighting and are ablated separately.

Finally, as secondary measures, we rerun the adversarial wrong-diagnosis prefix test from Disentangling Reasoning and Knowledge. There, Llama-3.1-8B-Instruct fell to 29.3% and BioMed-R1-8B held 56.4%. We also report ROUGE-L on MeQSum against the 53.15 BioBART-Large reports, to confirm the method does not trade robustness for generation quality.