Hospitals building models for rare diseases rarely have more than a few hundred patients confirmed by chart review. They do have thousands of "silver" labels: cheap, noisy guesses taken from billing codes or keyword rules. A recent method for rare-disease diagnosis, WEST, cleans these silver labels iteratively. A model trains on them, and its own predictions are then used to down-weight labels it disbelieves. The weakness is that the model judges its own homework. A high-capacity network trained on a small medical cohort readily learns shortcuts, such as a scanner tag, a hospital-specific code, or a billing habit, in place of the disease. When that same network is asked which labels to trust, it keeps the labels that agree with its shortcut and discards the ones that contradict it.
We propose to split the refinement step across two views of the same patient that are unlikely to share a shortcut. The first view is the structured record, meaning diagnosis, medication and procedure codes. The second is the free-text clinical notes. Each view is trained on silver labels that the other view has re-weighted. A label is flipped or dropped only when both views independently doubt it. Put in two sentences: iterative silver-label cleaning fails on small cohorts because the model can confirm its own spurious cues. Making two models with non-overlapping evidence check each other's labels should block that loop, and the test of the idea is whether this beats an equally sized two-model ensemble that sees the same evidence.
Core Claim
Agreement filtering is already known to matter. In Enhancing Semantic Role Labeling for Tweets Using Self-Training, two conditional random field taggers (classical sequence labelers) were trained on split partitions of the same data, and a tweet was pseudo-labeled only if both agreed confidently. Removing that agreement check dropped F1 from 51.7% to 41.6%, below the 48.3% no-self-training baseline. Those two labelers, however, saw the same kind of input. Our claim is narrower and more testable. In small clinical cohorts, the protection agreement provides comes mainly from view diversity, not from having two voters. Two models that see the same codes will share the same billing-code shortcut and agree on its errors. A codes model and a notes model mostly will not. So cross-view refinement should beat both single-model refinement (the WEST approach) and same-view two-model agreement, and the gap should widen as the gold set shrinks and as shortcut strength grows.
Relation to Prior Work
The closest work is A Weakly Supervised Transformer to Support Rare Disease Diagnosis from Electronic Health Records. WEST combines a large at-risk pool carrying silver labels with as few as 100 gold cases, encodes how often concepts recur, prunes codes using SapBERT embedding similarity, and lets a single Transformer's predictions re-weight silver labels during training. It reports AUC 0.93 / PPV 0.95 for pulmonary hypertension and AUC 0.87 / PPV 0.80 for severe asthma on private Boston Children's cohorts. We reuse its silver-plus-gold setup and its iterative re-weighting. We change who does the re-weighting.
The same paper also constrains our design. WEST notes that refinement only worked once feature pruning stopped the model latching onto noisy code co-occurrences. We read this as direct evidence of the self-confirmation risk. It also means our single-view baselines must include that pruning, or we would be beating a strawman.
Two papers supply the encoders for the two views. MiME builds patient embeddings around the visit, diagnosis and treatment hierarchy, with self-supervised code-reconstruction tasks, and gained most (15% relative PR-AUC) on its smallest data partition. It is a natural codes-view encoder for small cohorts. Ontology-driven weak supervision for clinical entity classification (Trove) shows that ontology-derived labeling rules, combined by an unsupervised label model and then used to train BioBERT, come within 1.3–4.9 F1 of fully supervised models. Trove supplies both our notes-view encoder recipe and our way of generating initial silver labels. Trove produces labels once and never revisits them, and it operates on text alone. We add iteration and a second modality.
Two further papers motivate the threat model. Unveiling COVID-19 from Chest X-ray with deep learning shows small-cohort models clustering by data source rather than pathology and collapsing externally. Learning Inter-Modal Correspondence and Phenotypes From Multi-Modal EHRs (cHITF) warns that modalities in one encounter are not uniformly linked. That warning matters here because it means the two views can disagree for legitimate reasons, not only because of label noise.
We deliberately do not take the knowledge-injection route of GraphCare. GraphCare reports that adding LLM-generated graphs let a model trained on 36 patients match baselines trained on 720. That result is real, but it addresses feature poverty, not label noise. It is orthogonal to our question and could be stacked on our method later.
Research Program
We start with a pool of patients, each carrying a silver label $\tilde y_i$ from labeling rules over both codes and notes, plus a gold subset of size $n_g$. We train a codes-view model $f_A$ and a notes-view model $f_B$.
At refinement round $t$, the weight that silver example $i$ receives when training $f_A$ is set by the notes model's belief:
$$w_i^{A,(t+1)} = p_{B}^{(t)}(\tilde y_i \mid x_i^{B}),$$
The notes model $f_B$ is weighted symmetrically by the codes model's belief. Here $x_i^A$ and $x_i^B$ are the patient's codes and notes, and $p^{(t)}$ is each model's calibrated probability after round $t$. Calibration is done by temperature scaling on the gold set.
A silver label is flipped only when both $p_A$ and $p_B$ assign the opposite class probability above a threshold $\tau$, which is tuned on held-out gold. Gold examples always keep weight one.
Patients on which the two views strongly disagree are logged as a separate disagreement set. We ask two questions of this set. First, does it concentrate genuine silver-label errors? Second, does spending part of the gold-annotation budget there beat random gold sampling? This secondary question links the method back to the annotation bottleneck. Even if refinement gains are small, a cheap way to choose which charts a clinician should review has value of its own.
To make the shortcut question measurable rather than anecdotal, we add a controlled shortcut injection. In the codes view only, we plant a synthetic marker code whose presence correlates with the silver label at a chosen strength, and with gold truth at chance. This mimics a hospital billing artifact. We then vary its strength.
Baselines
The naive reference point is gold-only training. The same codes-plus-notes architecture is fit to the $n_g$ gold patients alone, with silver labels ignored. Training on raw silver labels with no refinement is a second floor, since it shows how much damage the noise does. A flat XGBoost model over bag-of-codes trained on gold plus silver is included because WEST used it as a baseline and it is what many hospitals would actually deploy.
The strongest existing method is WEST-style single-model iterative refinement. One Transformer over concatenated codes and note features re-weights its own silver labels, with concept-frequency encoding and SapBERT-based code pruning enabled. On its own private data WEST reports AUC 0.93 and PPV 0.95 (pulmonary hypertension). On the MIMIC-III tasks we use, we found no reported result on Lacuna for WEST. We will therefore reimplement it. To make the comparison fair, it gets the same total parameter count as our two models combined, the same number of refinement rounds, and the same hyperparameter-search budget measured in trials.
For context on MIMIC-III in-hospital mortality, cHITF reports AUPRC 0.47–0.51 (cHITF). That figure comes from full-label training, so it is a ceiling reference rather than a competitor.
The ablation that isolates our key ingredient is same-view agreement. Two models of the combined architecture are trained on disjoint halves of the data, in the manner of the Twitter semantic role labeling paper, and cross-weight each other with the identical update rule. A second ablation is one-shot cross-view labeling, in which the two views' first-round beliefs fix the weights once and no further iteration happens.
Beating WEST-style refinement means a statistically reliable AUPRC gain (paired bootstrap over patients, five seeds) at $n_g \le 100$. Beating the same-view ablation by a clear margin is the result that supports the core claim. If cross-view beats WEST but ties same-view agreement, the gain comes from ensembling, not diversity, and we will report it that way.
Why not just…?
Why not just fine-tune a pretrained clinical language model on the gold set? With 25–100 gold patients, the evidence in the file is discouraging. In Domain Adaptation via Prompt Learning for Alzheimer's Detection, a generic prompt-tuning baseline stalled near 58–59% F1 while a structured method reached 0.8952. In the chest X-ray study, the deeper network overfit worse than the shallower one. Even so, this is the gold-only baseline, and we run it with a strong clinical encoder. If it matches our method at $n_g = 100$, then silver labels are not pulling their weight in that regime.
Why not just use two models that see the same data, as the Twitter self-training work did? This is the real threat, and it is the ablation the proposal is built around. If shared shortcuts are rare in practice, same-view agreement will work as well and our claim fails. The shortcut-injection experiment predicts where the methods should separate: same-view models should both absorb the planted code, while the notes model should not see it. If they do not separate there, the idea is wrong.
Why not just denoise silver labels once with a Trove-style label model and stop? Trove's label model beat majority voting and got close to supervised performance on entity tasks. However, it judges labels only through the labeling rules' agreement patterns, never through what a trained model learns about each patient. The one-shot cross-view ablation tests whether iteration adds anything beyond a good first denoising. If it does not, we will recommend the cheaper pipeline.
Why not just pool external labeled data from a large public database? Trove found that local weak supervision beat a model hand-labeled on external MIMIC-II notes by 2.3 F1. The Italian Twitter tagger in When silver glitters more than gold lost accuracy when the full 12,000-sentence external corpus was added. Pooling is therefore not a safe default, and we do not pursue it further.
Proposed Experiments
The main testbed is MIMIC-III, which has both codes and notes. We use three tasks: in-hospital mortality, 30-day readmission, and the 24-hour phenotype classification used by CTPD. We restrict each to rare phenotypes and subsample gold sets to 25, 50, 100 and 250 patients.
The task design involves a circularity that we handle explicitly. Mortality and readmission have recorded ground truth, so silver labels there come from imperfect code-and-keyword rules and are not circular. Phenotype labels in MIMIC-III are themselves derived from codes. For phenotypes, we will therefore generate silver labels from notes-only rules, and we will spend a small chart-review budget, a few hundred patients, on the gold evaluation set. Without that review, the phenotype results would test agreement with codes, not truth.
For external validity, the codes-view model is evaluated on eICU, where notes are largely absent. This also tests whether a model refined across two views still stands alone when only one view is available at deployment.
The decisive checks are three. First, the AUPRC curve versus $n_g$ for cross-view, same-view and WEST-style refinement. Second, under shortcut injection, the fraction of silver labels "corrected" toward the planted marker, and the gold-set AUPRC as marker strength rises. Third, on the disagreement set, the precision of view disagreement at finding true silver errors, together with the gain from spending 25 extra gold labels there rather than at random. A result where cross-view refinement holds its AUPRC under a strong planted shortcut while both same-view and single-model refinement degrade would be clear support. Equal degradation across all three would refute the idea.
References
- Enhancing Semantic Role Labeling for Tweets Using Self-Training
- A Weakly Supervised Transformer to Support Rare Disease Diagnosis from Electronic Health Records: Methods and Applications in Rare Pulmonary Disease
- MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
- Ontology-driven weak supervision for clinical entity classification in electronic health records
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Learning Inter-Modal Correspondence and Phenotypes from Multi-Modal Electronic Health Records
- GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs
- Domain Adaptation via Prompt Learning for Alzheimer's Detection
- When silver glitters more than gold: Bootstrapping an Italian part-of-speech tagger for Twitter
- CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis