Cross-Architecture Agreement as a Gate for Self-Labeling in Rare-Disease EHR Transformers

Models that find rare-disease patients in hospital records face two problems at once. Very few patients carry a confirmed diagnosis, and many undiagnosed patients sit in the "negative" pool. A recent answer is to train a transformer on a small set of expert-verified cases plus a large set of noisy, automatically derived ("silver") labels. The model then relabels the silver patients it is confident about and retrains on the result. This loop works, but it is a model grading its own homework. Any patient it gets wrong in round one becomes a confidently wrong training example in round two.

We propose to keep the transformer as the final predictor and change who decides when a silver label may be changed. A label is flipped only when a small committee of deliberately different models agrees. The committee contains transformers reading the record in different ways, a recurrent sequence model, and a gradient-boosted tree model over code counts. Patients on whom the committee disagrees are not relabeled. They are kept as "uncertain" and down-weighted. The bet is that models with different built-in assumptions make less correlated mistakes on rare, prodromal patients than copies of one transformer do. If so, agreement across architectures is a cheaper and more reliable filter for self-training than any single model's confidence.

Core Claim

The closest work is WEST. It trains a single weakly supervised transformer on roughly 100 gold labels plus silver EHR labels. It cleans the silver labels iteratively with its own predictions and reports AUC 0.93 and PPV 0.95 for pediatric pulmonary hypertension, and AUC 0.87 and PPV 0.80 for severe asthma. WEST has no mechanism against confirmation bias in that loop. RareBERT handles hidden positives with positive-unlabeled learning inside one transformer and reaches 80.1% AUPRC on X-linked hypophosphatemia. It also does not ensemble. Our claim is narrower and more testable than "ensembles help." On rare-disease phenotyping with weak labels, the errors of heterogeneous models are measurably less correlated than errors across transformer seeds. A relabeling rule that requires cross-architecture agreement therefore produces fewer wrong flips per round. That should translate into higher final AUPRC and better calibration than either self-refinement or a seed ensemble at matched compute.

Two other papers support this bet. Convolutional Nets Versus Vision Transformers for Diabetic Foot Ulcer Classification found that transformers lose to models with stronger built-in structure on small medical datasets, and won the DFUC 2021 challenge with a heterogeneous ensemble. That points to architectural diversity as a source of complementary errors. Hydra needed a 25-member ensemble to stabilize rare cell-type identification. Its members, however, are identical VAEs, so it does not test whether diversity matters. One file entry constrains the design. Comparing Methods for Uncertainty Estimation of Paraganglioma Growth Predictions shows that model combinations in a rare pathology can still be under-calibrated: its Bayesian approach covered only 78% of true values at a nominal 95% level. We therefore report calibration as a primary outcome, not an afterthought.

Research Program

Each refinement round trains $K$ committee members on the current labels. The members are:

  • a WEST-style transformer using concept-frequency encoding, which embeds how often each clinical concept occurs rather than their order;
  • an order-preserving transformer with type and temporal embeddings in the style of RareBERT;
  • a bidirectional LSTM over visits;
  • a gradient-boosted tree model over concept counts.

All members are scored on the gold set by cross-validation. A silver patient $i$ with current label $y_i$ is flipped to $1-y_i$ only if

$$\min_{k} \; p_k(1-y_i \mid x_i) > \tau,$$

where $p_k$ is member $k$'s calibrated probability (temperature-scaled on gold folds) and $\tau$ is a threshold. We pick $\tau$ so that the estimated flip precision on gold-masked patients stays above a target. Patients on whom members disagree beyond a margin receive a soft weight equal to one minus the spread of the members' probabilities. Unanimous high-confidence patients keep full weight.

After the final round, the transformer is retrained on the cleaned labels. We deploy it in two forms: the full committee, and a single transformer distilled from the committee's soft predictions. Reporting both addresses the inference-cost objection to ensembles.

We will also measure the quantity the method depends on: the pairwise correlation of member errors on held-out gold patients, computed separately for heterogeneous members and for transformer seeds. This is analysis, not just tuning. If heterogeneous errors are not less correlated, the core claim is false regardless of final scores.

Baselines

The naive reference point is the transformer trained on gold labels alone, without silver data. It shows whether weak supervision helps at all at each gold-label budget.

The strongest existing method is WEST's single-model iterative refinement, re-implemented on our cohorts with the same backbone, number of rounds, and hyperparameter search budget as our committee transformer. The reported figures for WEST are AUC 0.93 and PPV 0.95 on pulmonary hypertension at Boston Children's Hospital. Those cohorts are private, so the comparison will be our re-implementation on shared data, not those numbers. Beating WEST means higher AUPRC with a 95% bootstrap interval excluding zero at gold budgets of 100 or fewer, plus higher flip precision.

The second strong competitor is a five-seed ensemble of the WEST transformer using the same agreement rule. Its total compute matches the committee's, measured in GPU-hours. This is the critical ablation: it holds the gating rule fixed and removes only architectural diversity.

We will also run one variant that removes the boosted trees and one that removes the LSTM, to see which inductive bias contributes. Finally, we run a RareBERT-style positive-unlabeled transformer as the alternative way of handling hidden positives. For reference, it reports 80.1% AUPRC on XLH against 30.0% for Med-BERT. For the gated committee on public MIMIC cohorts, no reported result was found on Lacuna.

Why not just…?

Why not just use a seed or checkpoint ensemble of the transformer? This is the most serious threat. Seed ensembles are simpler and work in other settings, such as the checkpoint averaging in One Model to Pronounce Them All. If transformer seeds trained on the same biased silver labels fail on the same prodromal patients, agreement among them gives false comfort. The compute-matched seed-ensemble baseline and the error-correlation analysis settle this directly. If seeds match the heterogeneous committee in flip precision and AUPRC, we will report that the diversity premise fails and recommend the simpler method.

Why not just raise the single model's confidence threshold for flipping? A stricter threshold cuts wrong flips, but it also cuts correct ones. A single model is often confidently wrong on mimics. We will trace each method's flip-precision versus flips-made curve across thresholds. The committee is only worth it if its curve dominates the single model's, not if it merely flips less.

Why not just use gradient-boosted trees, given transformers' weakness in small data? In RareBERT, XGBoost reached about 43% AUPRC on XLH and lost more than 67% of its performance when 15% of events were deleted, compared with a 1.35% drop for RareBERT. Trees therefore look weak as the final predictor but may be valuable as a dissenting voter. Our tree-only refinement run and the leave-trees-out ablation measure both roles separately.

Why not just label more gold cases? For ultra-rare conditions, more expert-verified cases often do not exist. Still, the gold-budget sweep reports how many extra gold labels the committee is worth. If that number is small, the engineering is not justified.

Proposed Experiments

The public testbed is MIMIC-IV and eICU. We will define two or three low-prevalence conditions, for example pulmonary hypertension, which allows a direct analogue to WEST. Silver labels come from billing codes. Gold labels come from clinician review of 300 to 500 discharge notes per condition, split into training budgets of 25, 50, 100, and 200, with the remainder held out.

Two decisive checks follow. The first is controlled: we corrupt silver labels at known rates, including flipping gold positives to negative to mimic undiagnosed patients, then measure how many corruptions each method repairs versus introduces per round. The second is natural: we measure AUPRC, PPV at fixed recall, expected calibration error, and 90% interval coverage on held-out gold patients. Coverage is included because of the under-calibration seen in the paraganglioma work.

We will seek a collaboration to rerun on a WEST-style pediatric cohort and an adult multi-center cohort, the generalization WEST left open. The result we expect, and would publish either way, is a measured error-correlation gap between heterogeneous and same-architecture members, and whether that gap predicts the gain in refinement quality.