Co-training is a semi-supervised method in which two models look at the same example through different inputs, called "views," and each labels unlabeled data for the other. It works because the two models tend to make different mistakes. A radiology report has two such views built in: the "Findings" section, a detailed record of what the radiologist saw, and the "Impression" section, a short clinical conclusion. Co-training classifiers on these two sections already beats supervised training and self-training when labels are scarce. Separately, continuing to pre-train a language model on in-domain text (domain-specific pre-training) is the standard way to make clinical models better. The two practices pull against each other. Domain pre-training on whole reports teaches one encoder how Findings and Impressions relate, and so it may make the two views' errors move together, which is exactly what co-training relies on not happening.
We propose to test that conflict directly and to resolve it with view-specific continual pre-training. Instead of one clinical encoder, we pre-train two: one only on Findings text and one only on Impression text, drawn from large public report corpora. Each co-training branch then gets domain knowledge without sharing a representation with the other. The core claim is testable. Under a fixed pre-training compute budget, view-specific encoders will show lower cross-view error correlation and larger co-training gains than a single encoder pre-trained on full reports. They will also beat the domain-agnostic encoder used in prior co-training work. If the claim is wrong, the study still settles a practical question nobody has measured: whether domain pre-training and co-training add up or cancel.
Core Claim and Why It Is Not Already Done
The closest work is A Dual-View Approach to Classifying Radiology Reports by Co-Training. It trains two DistilBERT classifiers on Findings and Impression from 868 labeled and 10,000 unlabeled reports, with each passing confident, agreed pseudo-labels to the other. It reports that the co-trained Findings classifier gains 8 percentage points on tumor-aggressiveness classification over supervised Findings training, and that it beats self-training and a post-hoc ensemble. That paper motivates this proposal but never touches pre-training: both branches use a general-domain model. Featureless Domain-Specific Term Extraction with Minimal Labelled Data likewise gets its view diversity from architecture (a convolutional network versus a recurrent one) over static word vectors. Neither asks what happens to view diversity when the encoders become domain-specialized.
On the pre-training side, the relevant work either merges views or aligns them, and we argue both are wrong for co-training. Pre-training language model incorporating domain-specific heterogeneous knowledge into a unified representation flattens text, titles and knowledge triples into a single encoder. Dual-view Molecule Pre-training pre-trains separate sequence and graph branches but adds a consistency objective that pulls the two representations together. Our proposal borrows the separate-branch design from the molecule work and deliberately drops the alignment term. Our hypothesis is that alignment helps a fused representation but hurts co-training, because it makes the views agree in the wrong places. The broader observation in A Deep Fusion Model for Domain Adaptation in Phrase-based MT, that separately trained models keep source-specific patterns that pooling washes out, supplies the intuition for keeping the views apart during pre-training too.
Two findings constrain the design. First, co-training degrades when the initial views are noisy: BoolQ in Co-training Improves Prompt-based Learning for Large Language Models is the example. So view-specific encoders only help if each is individually strong enough, and splitting the pre-training data could weaken them. Second, Toward Expanding the Scope of Radiology Report Summarization to Multiple Anatomies and Modalities finds general BART-Large beating domain-pre-trained BioBART on radiology text. BERTHop, by contrast, reports a 6.46% gain from swapping BERT for the clinical BlueBERT. We therefore cannot assume domain pre-training helps, and we keep a general-domain arm throughout.
Research Program
We measure the quantity co-training theory cares about. Co-Training and Expansion shows that pseudo-labeling expands coverage safely when each view's confident region reaches examples where the other view is uncertain or wrong. A practical proxy is the class-conditional correlation of the two views' errors,
$$\rho = \mathrm{corr}\big(\mathbf{1}[h_F(x_F)\neq y],\ \mathbf{1}[h_I(x_I)\neq y]\ \big|\ y\big),$$
where $h_F$ and $h_I$ are the Findings and Impression classifiers, $x_F$ and $x_I$ are the two sections of a report, and $y$ is its gold label. The mechanistic prediction has two parts. Full-report pre-training raises $\rho$ relative to a general encoder. View-specific pre-training lowers it again while keeping per-view accuracy within a small margin of the shared encoder. Co-training gain should track low $\rho$ combined with adequate single-view accuracy.
The build is modest. Starting from one clinical-capable BERT-base-size checkpoint, we run continual masked-language-model pre-training (the model predicts hidden words from context) in four variants. The first is full reports, the standard practice. The second is Findings-only and Impression-only encoders, the proposal. The third is two full-report encoders from different random seeds, which isolates "two separately pre-trained models" from "view specialization." The fourth is full reports plus a cross-section contrastive alignment term in the style of the molecule work, which should raise $\rho$ if our account is right. All variants see the same total number of pre-training tokens. In the view-specific arm the two encoders split the budget, so each encoder sees fewer tokens than the single shared encoder. This handicap is deliberate: it makes a positive result hard to attribute to extra compute. A second setting gives each view encoder the full budget, to separate the data-splitting effect from the compute effect.
The pre-training corpus is the Findings/Impression pairs from MIMIC-CXR (MIMIC-CXR, 227,827 studies) and MIMIC-RRS (207,782 reports across three modalities and seven anatomies, from the summarization paper above), with the downstream test reports held out. We use an off-the-shelf clinical encoder as the starting point rather than training from scratch. We also record the token-splitting problem noted for BioBART, where technical terms get broken into poor subword pieces, as a shared confound across arms rather than a variable.
Downstream, each pre-training variant feeds the same co-training loop as Han et al.: agreement plus confidence filtering, a swept top-$k\%$ pseudo-label threshold (their best values were 50% and 25% on their two tasks), and averaged-ensemble inference. Every arm gets the same threshold search.
Baselines
The naive reference points are two. The first is a supervised classifier on the labeled seed set using the concatenated report with a general encoder. The second is the CheXpert rule labeler, which needs no training labels at all. On the 687 manually annotated MIMIC-CXR gold reports, the MIMIC-CXR paper reports CheXpert-labeler F1 of 0.973 for pleural effusion and 0.956 for pneumonia. On common findings, then, a learned low-label method must approach these numbers to matter, and our main claims will be made on the rarer and more hedged labels where rules are weaker.
The strongest existing method is Han et al.'s dual-view co-training, rerun with (a) its original general-domain DistilBERT-class encoder and (b) the standard full-report domain-pre-trained encoder shared by both branches. Variant (b) is the key competitor, since it is what a practitioner would do by default. That paper reports only relative improvements (8 points on Aggressiveness over supervised Findings), and no absolute accuracies were found on Lacuna. We therefore reimplement it on our splits rather than compare against a quoted score. It runs with identical seed labels, unlabeled pool, fine-tuning epochs and threshold sweep.
We also run self-training with the domain encoder on full reports, which the Han et al. comparison favored co-training over, but only with a general model. We run a supervised ensemble of separately trained Findings and Impression classifiers, and a supervised classifier on the domain encoder, which tests whether domain pre-training alone closes the gap. The two ablations that isolate our ingredient are the seed-diverse full-report encoders and the contrastively aligned encoder. Beating the baselines means the view-specific arm improves macro-F1 over shared-encoder co-training by more than the across-seed standard deviation, across at least three of the four label budgets, with five fine-tuning seeds per cell. Han et al. ran once, which is a limitation we fix.
Why not just…?
Why not just use one domain-pre-trained encoder for both branches? This is the default and the main threat. If full-report pre-training does not in fact raise $\rho$, view-specific pre-training has nothing to fix, and the split data may only weaken each branch. The shared-encoder arm with measured $\rho$ settles it directly. We commit to reporting it even if it wins.
Why not just skip co-training, since domain pre-training already gives large gains? BERTHop's 6.46% gain from BlueBERT suggests domain pre-training alone might absorb most of the benefit. The supervised domain-encoder arm at each label budget measures the remaining headroom. If co-training adds nothing on top at 868 labels, we will say so and report where, if anywhere, in the smaller budgets it still helps.
Why not just get diversity from two random seeds or two architectures? Seed and architecture diversity is cheap and worked for term extraction in the CNN-versus-LSTM paper. The seed-diverse full-report arm tests whether view-specific data matters beyond having two independently pre-trained models. If it matches the view-specific arm, the conclusion shrinks to "pre-train twice," which is still useful but a smaller claim.
Why not just align the sections contrastively, as dual-view molecule pre-training does? Alignment helps when the goal is one fused representation. Our hypothesis is that it hurts co-training by correlating errors. The aligned arm is a falsification test: if it raises co-training gains, our mechanism is wrong.
Proposed Experiments
The primary benchmark is MIMIC-CXR report classification over its 14 pathology labels. We use seed budgets of 50, 200, 868 (matching Han et al.) and 2,000 labeled reports, an unlabeled pool of 10,000 and 50,000 reports, and testing on the 687 gold-annotated reports. The labels include negated and uncertain mentions, which is where the two views should disagree most. The metrics are macro-F1 and per-label F1, plus $\rho$ and single-view accuracy for every arm. If the Alberta Health Services data used by Han et al. can be accessed, we replicate on its tumor and aggressiveness tasks for a direct comparison. For a transfer check beyond chest X-ray, we build classification labels on the non-chest MIMIC-RRS modality-anatomy pairs and test whether view-specific encoders pre-trained mostly on chest text still help on CT and MRI. The informal-text gap seen for BioBART makes this a real risk.
Four checks are decisive. First, $\rho$ must be ordered as predicted (aligned ≥ full-report ≥ view-specific). Second, across all arms and budgets, co-training gain must correlate negatively with $\rho$ once single-view accuracy is controlled for. Third, the view-specific arm must beat the shared-encoder arm under the compute-matched budget, not only the doubled one. Fourth, the gain must exceed the seed-diverse ablation. If the first check holds and the third fails, the result is still informative: domain pre-training does couple the views, but the loss in per-view strength from splitting the data outweighs the gain. That tells practitioners to choose between the two methods rather than stack them.
References
- A Dual-View Approach to Classifying Radiology Reports by Co-Training
- Featureless Domain-Specific Term Extraction with Minimal Labelled Data
- Pre-training language model incorporating domain-specific heterogeneous knowledge into a unified representation
- Dual-view Molecule Pre-training
- A Deep Fusion Model for Domain Adaptation in Phrase-based MT
- Co-training Improves Prompt-based Learning for Large Language Models
- Toward Expanding the Scope of Radiology Report Summarization to Multiple Anatomies and Modalities
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
- Co-Training and Expansion: Towards Bridging Theory and Practice
- MIMIC-CXR: A large publicly available database of labeled chest radiographs