Small language models fine-tuned on one field, such as medicine, law, or hardware design, now match much larger general models on that field's benchmarks. Those benchmarks, though, ask each question in one fixed wording, usually the terse register of an exam or a specialist. Real users of a clinical or legal assistant write in many registers. A nurse writes "MI," a patient writes "heart attack," and a physician writes "myocardial infarction." Separately, a large literature shows that language models change their answers when a question is reworded without changing its meaning. Nobody has measured whether specialization makes this worse. This proposal asks that question directly and tests one specific hypothesis: fine-tuning on narrow expert text makes a model fragile to shifts in vocabulary and register (lay versus expert wording, abbreviations versus full terms, synonyms from the domain's terminology) far more than to generic noise such as typos.
In two sentences: we take one base model, specialize it on medical, legal, and Chinese medical data, and track how its answers change under meaning-preserving rewrites as training proceeds, separating typo-style noise from domain-register shifts and scoring with methods that do not mistake valid paraphrased answers for errors. If register brittleness grows with specialization, we then test whether the fix lies in which rewrites the model sees during training or in how it is trained on them. Our starting assumption, which the experiments are designed to overturn if wrong, is that the rewrite distribution matters more than the training objective.
Core Claim
The claim has two parts. The first is a measurement claim. Relative to its own base model, a domain-specialized compact model shows a widening gap in answer consistency under domain-register perturbations, and that gap is larger than the gap under generic perturbations. Let $S_f(m)$ be the fraction of question groups on which model $m$ gives semantically equivalent answers across all rewrites from perturbation family $f$. We predict
$$ \Delta_f = S_f(\text{base}) - S_f(\text{specialized}), \qquad \Delta_{\text{register}} > \Delta_{\text{typo}} \ge 0, $$
with $\Delta_{\text{register}}$ growing over fine-tuning steps. The intuition is that fine-tuning concentrates capacity on the exact surface forms found in expert corpora, which narrows the set of phrasings the model treats as equivalent.
The second is a mitigation claim. If this brittleness exists, it can be largely removed by exposing the model to register rewrites drawn from the domain's own terminology during specialization. How much a consistency-specific objective adds beyond that data should be tested, not assumed.
Two strands of prior work motivate this. PAFT: Prompt-Agnostic Fine-Tuning shows that fine-tuning on fixed instruction templates overfits to surface wording, and it fixes this by sampling from 400+ generated templates. Its variations are instruction wrappers on general benchmarks, not the domain vocabulary inside the question. Specializing Smaller Language Models towards Multi-Step Reasoning shows that specialization imposes a "tax": math-distilled Flan-T5 loses general reasoning on BigBench Hard and even loses few-shot ability when trained only on zero-shot format. It never measures phrasing sensitivity. The gap our measurement fills is that no work in the file compares a specialist's phrasing sensitivity with that of its own base model.
One paper constrains the design. Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs argues that much measured sensitivity is an artifact of string-matching scoring. On ARC-Challenge, Gemma's accuracy spread across 12 templates had a standard deviation of 0.28 under heuristics but 0.005 under an LLM judge. Any brittleness we report must therefore survive meaning-aware scoring, or it is not a finding.
Research Program
Stage 1: the specialization trajectory. We fine-tune Llama-3.1-8B-Instruct, the compact model used by both EC-RAFT and Con-RAG, into three specialists:
- a biomedical model trained on the MultiMedQA training splits;
- a Chinese medical model trained on CMExam training data, following the recipe of Benchmarking Large Language Models on CMExam;
- a legal model trained on CaseHOLD and related data, using the benchmark set from Compressing Language Models for Specialized Domains.
We save checkpoints along training so that consistency can be plotted against how far specialization has gone, not just compared at the endpoints. We also run the published EC-RAFT recipe on ClinicalTrials.gov eligibility-criteria generation. That setting is open-ended generation, where a dropped exclusion criterion is a real change in meaning.
Stage 2: a two-family perturbation suite.
- Generic family. This follows the taxonomy of Investigating the Robustness of Retrieval-Augmented Generation at the Query Level: character and word typos at 10% and 25%, redundancy insertion, formal tone, and ambiguity. We drop ambiguity from the consistency metric because it changes meaning by design.
- Register family. This is new: lay-language rewrites of expert questions, abbreviation contraction and expansion, synonym substitution within domain terminology lists, and conversion between exam-stem and free-form case-narrative format.
Every rewrite must be produced independently by two different rewriter models and pass an equivalence check. On a stratified sample of several hundred items per domain, domain annotators confirm that the rewrite preserves the answer. This prevents register rewrites from silently changing the clinical or legal content.
Stage 3: scoring that survives the artifact critique. For multiple-choice tasks we report the choice-agreement rate within a question group and worst-case group accuracy. Letter-choice agreement is not vulnerable to the string-matching artifact. For open-ended outputs we report an LLM-judge equivalence score alongside an embedding-similarity score and exact match, so readers can see whether conclusions depend on the scorer. For the retrieval-augmented BioASQ setting we adopt the decomposed metrics of Con-RAG: retriever consistency (overlap of retrieved documents across rewrites), generator consistency (with retrieved context held fixed), and end-to-end consistency. This lets us test whether specialization moves brittleness into the generator. Investigating the Robustness of Retrieval-Augmented Generation at the Query Level hinted at exactly this on BioASQ, where answers degraded even when recall stayed stable.
Stage 4: separating data from objective. We compare specialization variants that all use an identical budget of examples and optimizer steps:
- plain specialization;
- specialization with generic rewrites;
- specialization with register rewrites;
- register rewrites plus a group-consistency reward in the style of PS-GRPO, combined with the correctness reward.
This design tells us whether the objective matters once the right rewrite distribution is present.
Baselines
Do-nothing references. The first is the unspecialized base model. It anchors $\Delta_f$ and tells us whether specialization helps or hurts consistency at all. The second is the plain specialist trained on fixed benchmark wording, which is the object under study. Both are evaluated on identical rewrite sets with the same decoding settings and several sampled answers per rewrite, so that sampling noise is separated from rewrite sensitivity. We will also report each specialist's clean-wording accuracy against published anchors: fine-tuned ChatGLM-6B at 45.3% on CMExam (from the CMExam paper), and BioBERT at 68.1% on PubMedQA (from PubMedQA). This confirms our specialists are competitive enough for the question to matter.
PS-GRPO / Con-RAG. The strongest existing consistency method is PS-GRPO from Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewards. It samples several paraphrases of each query, rolls out several answers per paraphrase, and rewards agreement across the group. Its authors report end-to-end consistency on TriviaQA rising from 77.8% to 91.3% with Llama-3.1-8B. It is the strongest competitor because it directly optimizes the quantity we measure. We run it on our domain training data with its own general-purpose paraphrases, the same base model, and the same number of optimizer updates as our register variant, with equal hyperparameter search budgets. No reported result was found on Lacuna for PS-GRPO on any biomedical, legal, or Chinese medical benchmark. We count as beating it any case where register-aware training matches its clean accuracy while exceeding its register-family consistency by more than the seed-to-seed spread.
PAFT. PAFT reports 7% higher accuracy on unseen prompts using dynamic template sampling. We run it with a comparable domain-adapted template pool, again at the same step budget. It tests whether wrapper-level variation already covers register variation.
Key ablation. The ablation that isolates our main ingredient swaps register rewrites for an equal number of generic rewrites while holding everything else fixed.
Why not just…?
Why not just rewrite every incoming query into canonical expert wording before it reaches the specialist? This is the most serious threat, because a strong general model may map "heart attack" to "myocardial infarction" quite reliably. Some evidence cuts against it. The query-level robustness study found that LLM-generated rewrites such as formal-tone conversion often degraded retrieval and generation instead of stabilizing them, and rewriting adds a second generative step that can itself vary with phrasing (Rewrite-to-Rank). That evidence comes from general domains, so we test the alternative directly. We put a normalizing rewriter, the same size as our specialist and also a larger one, in front of the plain specialist and measure the same consistency and accuracy metrics plus added latency. If normalization closes the register gap at acceptable cost, we will report that as the practical answer. The measurement contribution stands either way.
Why not just augment the fine-tuning data with paraphrases and use ordinary supervised training? This is not a competing idea but one of our conditions. Con-RAG found plain supervised fine-tuning underperformed group rewards on general question answering. However, their paraphrases were not register-targeted, so we do not assume that result transfers. If register-augmented supervised training matches register-plus-consistency-reward training, we conclude that data coverage is what matters and recommend the cheaper method.
Why not just use a larger general model zero-shot? Larger models lose on domain accuracy in the file: Gemini-1.5-Flash scored 82.18 BERTScore against EC-RAFT's 86.23, and GPT-4 reached 61.6% on CMExam against humans' 71.6%. Being large also does not guarantee consistency; Automating the loop in traffic incident management on highway found rephrased incident descriptions changed large models' recommended responses. We include one large general model as a reference point on consistency. If it is both more consistent and close in accuracy, that weakens the case for specialization under register shift, and we will say so.
Proposed Experiments and Decisive Checks
The first decisive check is whether $\Delta_{\text{register}}$ exceeds $\Delta_{\text{typo}}$ under the LLM-judge and choice-agreement scorers, with confidence intervals from bootstrap resampling over question groups, in at least two of the three domains. If register brittleness appears only under string matching, the hypothesis fails in the way Flaw or Artifact predicts, and we report that as the result.
The second is whether the consistency gap grows monotonically along the checkpoint trajectory. A monotone trend ties the brittleness to specialization rather than to a quirk of the endpoint model.
The third, on BioASQ with a fixed retriever, is whether specialization lowers generator consistency while leaving retriever consistency unchanged. This would reconcile the file's apparent contradiction between generator-dominated and retriever-dominated failure: the balance shifts depending on how specialized the generator is.
The fourth is the data-versus-objective comparison in Stage 4, reported with clean accuracy alongside consistency. We add EC-RAFT criteria generation as a stress test in which we count omitted or altered exclusion criteria across rewrites of the same trial summary. A consistency gain that also shrinks the base model's lost general ability on held-out general benchmarks would indicate that register exposure partly offsets the specialization tax. We treat that as a secondary outcome, not a promise.
References
- PAFT: Prompt-Agnostic Fine-Tuning
- Specializing Smaller Language Models towards Multi-Step Reasoning
- Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
- EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria through Retrieval-Augmented Fine-Tuning
- Improving Consistency in Retrieval Augmented Systems with Group Similarity Reward
- Benchmarking Large Language Models on CMExam - A comprehensive Chinese Medical Exam Dataset
- Compressing Language Models for Specialized Domains
- Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
- PubMedQA: A Dataset for Biomedical Research Question Answering
- Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewriting
- Automating the loop in traffic incident management on highway