Is Domain Pre-Training Worth More as a Second Opinion?

Training a language model further on specialist text, such as medical abstracts or legal contracts, is usually judged by one question: does the specialised model beat the general one? The answer is often messy. On the Chinese biomedical benchmark CBLUE, medical models do not consistently beat a general RoBERTa, particularly on colloquial patient queries. On legal contract review, 8GB of extra legal text bought only about 3%. We propose to ask a different question. Does domain pre-training produce a model that makes different mistakes from a general model, so that the two combined (an ensemble, which averages or weights their predictions) beat any equally sized committee built from cheaper sources of variety? Those cheaper sources include different random seeds, different model families, or different input preprocessing.

The study is built to resist the hidden degrees of freedom that make such comparisons unreliable. It uses frozen splits, validation-only weight selection, equal tuning budgets, and a controlled model pair in which the only difference is the pre-training corpus. It uses expert-annotated benchmarks, where domain knowledge actually matters, and it adds a small multi-expert re-annotation to test a second claim. That claim is that disagreement between a domain model and a general model lands on the items where human experts also disagree. If both claims hold, practitioners get a concrete rule: spend domain pre-training compute on a second opinion, not a replacement. If they fail, the field learns that the diversity attributed to domain pre-training is no better than a change of random seed.

Core Claim

Under compute-matched and protocol-matched conditions, a two-member ensemble of a general model and its domain-continued twin will have a lower "double-fault" rate than two-member ensembles built from seed variation, preprocessing variation, or cross-family variation. The double-fault rate is the fraction of test items both members get wrong. The ensemble gain should concentrate on items written in a register the domain corpus does not cover, such as colloquial patient queries. On items where experts disagree, the members' disagreement should rise more than it does for the other ensemble types.

The closest existing work already ensembles domain and general models, so the novelty is not "ensemble them." Technical Report on Shared Task in DialDoc21 pairs a domain-adapted RoBERTa with a deliberately general ELECTRA and reports a gain of over 4% in F1 and Exact Match. However, it changes the architecture, the pre-training objective, and the corpus all at once, and it tunes ensemble weights on the validation set it reports. Biomedical Question Answering with Transformer Ensembles ensembles two biomedical models, BioBERT and BioM-ELECTRA. Its grid search places 0.996 of the weight on one member, which suggests that two domain models offer almost no complementary signal. That paper motivates our design: if diversity comes from the corpus, a domain-plus-domain pair should be weak, and a domain-plus-general pair should be strong. Neither paper isolates the corpus as the cause, and neither compares against cheaper diversity sources at equal size.

Research Program

The central object is a twin pair: one general base model, and a copy given continued pre-training on domain text. A third copy receives continued pre-training on an equal number of general tokens. This third copy separates "domain text" from "simply more training." Because the twins share architecture and tokenizer, they also permit weight averaging, which gives us a strong one-model alternative to ensembling.

We build twins in two domains with expert-annotated benchmarks. For Chinese biomedicine, we use a RoBERTa-style base continued on medical text and evaluated on CBLUE. That benchmark was annotated by 3 to 5 physicians and medical students per sample with Fleiss' Kappa of 0.9. It also mixes formal tasks (entity recognition, relation extraction) with colloquial search-query tasks (KUAKE-QIC, KUAKE-QTR), which gives the register-split test a natural home. For English biomedicine, we use a BERT-style base continued on PubMed text and evaluated on PubMedQA, where two physicians labelled 1,000 yes/no/maybe items. As a third, low-resource-of-labels domain, we use the legal contract benchmark CUAD, whose authors already showed that legal pre-training alone helps little. That makes CUAD the domain where the "second opinion" reading is most informative.

The comparison set fixes ensemble size at two members and varies only the source of diversity. The options are two seeds of the general model, two seeds of the domain twin, the general model paired with its domain twin, the general model paired with the general-continued control, and the general model paired with a different model family of similar size. That last pairing is the text-classification analogue of the "jury of peers" in Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification. That paper showed cross-family disagreement beats single-model resampling for detecting errors, but only on general-domain tasks.

Following the protocol lessons of Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline, every member receives the same hyperparameter search budget. That paper showed that controlled training recipes moved PointNet++ from 90.0% to 93.3% on ModelNet40, erasing claimed architectural gains. The protocol also follows Empirical Study of Transformers for Source Code, which showed a simple attention variant matched complex tree methods once splits were cleaned. Checkpoints and ensemble weights are chosen on a held-out slice carved from the training data. The official test split is read once. All numbers are reported over at least five seeds with intervals.

The expert-annotation component is deliberately small. Standard releases collapse labels to a single answer, a loss of information that EDUE argues destroys useful uncertainty signal. In medical imaging, EDUE trained models to match multi-rater variance. We will first check whether per-annotator labels exist for CBLUE or PubMedQA. If they do not, we will re-annotate roughly 300 PubMedQA test items and 300 items from colloquial and formal CBLUE tasks, with three clinicians each, labelling independently. Annotators will see no model predictions, because Does AI Assistance Preserve or Collapse Disagreement? found that pre-annotations shifted agreement and boundary placement. That study was conducted with non-experts, so we do not assume its reassuring entropy result transfers. This gives each item an expert-disagreement score. We then ask how well each ensemble's member disagreement predicts it, measured by the area under the ROC curve.

Baselines

The naive reference point is the better single member of each pair, trained under our protocol. On PubMedQA, the file's published reference points are BioBERT at 68.1% accuracy with a multi-phase pipeline, a majority-class guess at 55.2%, and human experts at 78.0% (PubMedQA). On CBLUE, the strongest reported single model is the general RoBERTa-wwm-ext-large, with an average score of 70.0 against a human average of 77.1 (CBLUE). On CUAD, the strongest reported model is DeBERTa-xlarge, with an area under the precision-recall curve of 47.8% and precision at 80% recall of 44.0% (CUAD). We will reproduce these under our protocol rather than copy them. A twin-pair ensemble counts as useful only if it beats the reproduced best single member by more than the seed-to-seed interval.

The strongest existing ensemble method is the validation-optimised weighted ensemble used in DialDoc21 and in Biomedical Question Answering with Transformer Ensembles. The latter reports a Factoid mean reciprocal rank of 0.6969 and a List F1 of 0.5374 on BioASQ 11b. We apply the same weight search to every pair type, using the held-out training slice. This makes the comparison about the members, not the combiner.

The seed ensemble and the cross-family ensemble are the main competitors for the core claim. They receive identical member budgets, so any advantage of the twin pair must come from the pre-training corpus.

Weight averaging of the twins is included because, as FusionBench documents, fusion in parameter space is a standard single-model alternative to ensembling. FusionBench also found weight averaging weaker than input-routed mixing on its general-domain tasks.

The general-continued control is the ablation that isolates the key ingredient. If pairing the general model with this control gives the same double-fault rate as pairing it with the domain twin, the corpus is not doing the work.

Why not just…?

Why not just ensemble several random seeds of the best single model? Seed ensembles are cheap and often strong. If they match the twin pair's double-fault rate, our claim is simply false. The weak case for seeds comes indirectly from the 0.996 weighting in Biomedical Question Answering with Transformer Ensembles, where two similar domain models were nearly redundant. That paper is not a seed experiment, though, so the matched seed ensemble in our design is the real test.

Why not just ensemble general models from different families? This is the most serious threat. The jury-of-peers result in Complementing Self-Consistency with Cross-Model Disagreement shows that family diversity already beats single-model resampling, and it requires no pre-training compute. Against it, the same paper's open-problems discussion notes that families trained on similar web corpora share blind spots, so their disagreement can collapse. Domain text is exactly what web corpora under-represent. Our cross-family pair settles the question directly. If it matches the twin pair on the expert-labelled benchmarks, we will report that domain pre-training is not worth its cost as a diversity source.

Why not just average the twins' weights and serve one model? Weight averaging removes the doubled inference cost, which is the main operational objection to ensembles. We test it as a baseline. If it recovers most of the ensemble gain, the practical recommendation becomes "continue pre-training, then merge," and that is still a finding about domain pre-training as complementary signal. We expect it to fall short on colloquial tasks, because averaging should blur register-specific behaviour that FusionBench found better preserved by routing than by averaging. That expectation remains a hypothesis.

Why not just tune the domain model better? The point-cloud and source-code re-evaluations show that careful tuning can erase apparent gaps. That is why every member, including the single-model reference, gets the same tuning budget. Any ensemble gain that disappears under that budget will be reported as a protocol artefact.

Proposed Experiments and Decisive Checks

The primary measurement is the double-fault rate and the ensemble gain over the best member, for each pair type and benchmark. The claim survives only if the twin pair beats both the seed pair and the cross-family pair beyond seed intervals, on at least two of the three benchmarks.

The second check is register-stratified. On CBLUE, the twin-pair gain should be larger on KUAKE-QIC and KUAKE-QTR than on the formal tasks. The reasoning is that CBLUE and BioBART both report that formal-literature pre-training transfers poorly to informal medical language, which is exactly where a general member should cover for the domain twin. A flat gain across registers would undercut the explanation even if the overall gain holds.

The third check uses the re-annotated items. Twin-pair disagreement should predict expert disagreement better than seed-pair or cross-family disagreement does. Its accuracy on items where experts agree should be reported separately, so that a model cannot "win" simply by being uncertain everywhere.

Finally, we report a compute ledger: accuracy against total training and inference cost for every configuration. This shows whether the tokens spent on domain pre-training buy more than the same tokens spent training a third general member.