When two classifiers disagree, standard practice counts the disagreement and ignores what it is about. A convolutional network that calls a radiograph "edema" and a vision transformer that calls it "consolidation" are scored as disagreeing exactly as much as if the second had said "pneumothorax." A clinician would call the first case a near-miss between related findings and the second a warning sign. We propose to score disagreement between two architecturally different image encoders with a distance between label meanings. Both encoders are trained to map chest X-rays into a shared space defined by a medical language model's embeddings of written disease descriptions. The distance between two diseases is then the distance between their descriptions in that space. Because the space is defined by text, this works for diseases neither model was trained to recognize. Index-level uncertainty methods cannot offer that.
In two sentences: train a CNN and a ViT to match X-rays against medical-language-model embeddings of disease definitions, and flag a case as risky when the two models' predictions sit far apart in label-meaning space rather than when they merely differ. We test whether this flags clinically serious errors, especially on unseen diseases, better than ordinary ensemble disagreement, seed ensembles, or single-model confidence.
Core Claim
Our claim concerns uncertainty, not accuracy. Two encoders with different built-in assumptions produce errors that overlap less than two copies of the same model do. The CNN is biased toward local texture and the transformer toward global context. If their predictions are compared through a cost that reflects how related two findings are, their disagreement becomes a better predictor of severe mistakes than index-level disagreement. The advantage should be largest on diseases held out from training.
This is distinct from three nearby lines of work.
Heterogeneous ensembles show that architectural diversity improves calibration and selective prediction, but they operate over fixed label indices. Asymmetric Duos pairs a large model with a small, structurally different "sidekick" and fuses their logits with learned temperatures on ImageNet-style benchmarks. Transformer-CNN Cohort uses a ViT and a CNN to correct each other's pseudo-labels in segmentation. Both motivate our choice of encoder pair. Neither has label semantics, zero-shot classes, or clinical data.
Zero-shot chest X-ray models ground labels in domain language but use one image backbone and report no uncertainty. CXR-ML-GZSL projects DenseNet-121 features and BioBERT disease embeddings into a shared space with ranking, alignment and consistency losses. BoMD anchors local visual descriptors to BlueBERT embeddings to detect noisy training labels. The CLIP fine-tuning study aligns radiographs with report sentences and relaxes the contrastive loss. Its motivation is that multi-label studies sharing a finding get treated as false negatives.
Uncertainty work on model disagreement, such as cross-model disagreement for uncertainty and Ising-model label aggregation, models dependence between predictors. It still treats labels as unrelated symbols.
One claim in our own background material needs correcting. It is tempting to say that logit ensembling is impossible for unseen classes. It is not. If both encoders score images against the same label-text embeddings, their per-class similarity scores can simply be averaged, even for new classes. We therefore do not claim that fusion requires a new latent-space method. We use plain temperature-weighted score averaging for prediction. The only new ingredient is how disagreement is measured.
Research Program
Label space. Each of the 14 ChestX-ray14 findings gets a short written description in the style of label-description training, which showed that training against definitions rather than bare names makes zero-shot text classification more accurate and less sensitive to prompt wording. The descriptions combine a definition, typical radiographic appearance, and common synonyms. They are encoded once by a frozen biomedical language model, PubMedBERT or BlueBERT. Domain pre-training earns its place here: BERTHop reports a 6.46% AUC gain on OpenI from swapping BERT for BlueBERT. We expect domain text to place clinically related findings closer together than a general model does, and we test that expectation directly.
Encoders and training. Two image encoders, a DenseNet-121 and a ViT of comparable compute, each get a projection head into the label space. They are trained separately on seen classes with the CXR-ML-GZSL objective. Images are scored against label embeddings rather than against other patients' reports, which also avoids the instance-instance false-negative problem raised by the CLIP fine-tuning study.
Semantic disagreement score. Each encoder $m \in \{A, B\}$ produces per-label probabilities over the $K$ findings. We normalize these into a distribution $q_m$ over labels. The disagreement score is the optimal-transport cost of moving one model's label mass onto the other's:
$$U(x) = \min_{\pi \in \Pi(q_A, q_B)} \sum_{i,j} \pi_{ij}\, C_{ij}, \qquad C_{ij} = 1 - \cos(t_i, t_j),$$
where $\pi$ is a transport plan whose marginals are $q_A$ and $q_B$, and $t_i$ is the language-model embedding of label $i$'s description.
Because the per-label probabilities do not sum to one in multi-label prediction, we also report a variant that adds the difference in total predicted mass as a separate term. This keeps disagreement about how many findings are present from being normalized away. When $C_{ij}$ is constant off the diagonal, $U$ reduces to ordinary total-variation disagreement, so the index-level baseline is a special case of the proposal.
Analysis. Beyond the headline test, we check whether the CNN–ViT disagreement is structured the way the architecture story predicts. Specifically, we ask whether the two models diverge more on focal findings (nodule, mass) versus diffuse ones (edema, cardiomegaly). This is descriptive, not a claim we depend on.
Baselines
The naive reference is single-model confidence. We use the maximum per-label probability, or the mean entropy across labels, of the better single encoder as the abstention signal, with no ensemble at all. Any gain from a second model must first clear this.
The strongest existing accuracy baseline is CXR-ML-GZSL on the ChestX-ray14 split with 10 seen and 4 unseen classes. It reports an unseen-class AUROC of 0.66 and a seen/unseen harmonic-mean AUROC of 0.72. Our single DenseNet encoder is a reimplementation of it and should match these numbers before anything else is claimed. Beating it on accuracy is not our goal. We only require that fusing the two encoders does not fall below it.
For in-distribution CheXpert, the strongest reported number is a mean AUC of 0.900 on the five competition pathologies from the CLIP fine-tuning study. We cite it as context, not as a head-to-head target, because its training data differs.
The strongest uncertainty competitor is an Asymmetric Duos–style ensemble (Asymmetric Duos). It uses the same two encoders and the same learned-temperature fusion, but scores disagreement at the index level, as total variation or the fused prediction's entropy. This baseline isolates our key ingredient, because everything is identical except the cost matrix. We found no reported result on Lacuna for heterogeneous-ensemble selective classification on chest X-rays, so we run it ourselves.
The homogeneous control is a seed ensemble: two DenseNets, or two ViTs, of matched total compute. It is scored with both the semantic and the index disagreement measures. This separates the value of architectural diversity from the value of the semantic cost.
Cost-matrix ablations test where the semantics should come from. We compare the domain language model, a general-domain BERT encoding of the same descriptions, the domain model on bare label names without descriptions, a random cost matrix with matched statistics, and a cost built from label co-occurrence in the training set.
All methods share the same training data (ChestX-ray14 seen classes, plus MIMIC-CXR pre-training where used), the same descriptions, and the same hyperparameter search budget per encoder. Temperatures and thresholds are tuned on a held-out validation split and never on test data.
We count the baseline as beaten under two conditions. First, the semantic score must lower the area under the risk–coverage curve relative to index disagreement on unseen classes, with non-overlapping bootstrap confidence intervals over patients. Second, it must lower the rate of severe errors retained at 80% coverage.
Why not just…?
Why not just average the two models' scores and use ordinary disagreement? This is the main competitor, and it is exactly our index-level baseline. Averaging handles prediction on unseen classes perfectly well, which is why we adopt it for fusion. The open question is only whether "which labels they disagree on" carries information beyond "how much they disagree." If the semantic score does not beat total variation on the unseen split, the central claim is false, and we will report it as such.
Why not just ensemble seeds of the best single model? Asymmetric Duos reports that a heterogeneous pair matched a five-model deep ensemble at roughly a quarter of the compute on ImageNet-scale benchmarks. Transformer-CNN Cohort shows that identical architectures reinforce each other's errors. Neither result is from chest X-rays, though. Our matched-compute seed ensemble, scored with the same semantic measure, settles whether diversity matters here. If homogeneous pairs do as well, the contribution shrinks to the cost matrix alone.
Why not just use label co-occurrence or a hand-built ontology instead of a language model? This is a real threat on seen classes. Co-occurrence directly encodes which findings travel together and may beat embedding distance there. Its limit is that co-occurrence statistics do not exist for a disease absent from training, whereas a description can always be written and embedded. If co-occurrence ties or wins on seen classes, we will say so. The domain model's role would then narrow to extending structured disagreement to unseen classes, which is the regime CXR-ML-GZSL targets.
Why not just prompt a large general vision-language model? Evidence in adjacent domains argues against assuming scale wins. DermFM-Zero, a 304M-parameter dermatology model, outperformed DINOv3-7B with a balanced accuracy of 0.744. That is dermatology, not radiology, and it does not address uncertainty, so we treat it as motivation rather than proof. The general-BERT cost ablation tests the narrower version of this question that matters for us.
Proposed Experiments
Training and evaluation data. We train on the ChestX-ray14 seen classes. We evaluate on that dataset's unseen split, and on OpenI and CheXpert for transfer. BoMD reports 89.57% mean AUC for NIH-to-OpenI transfer, which serves as a sanity reference for transfer accuracy.
Metrics. We report per-label AUROC, the harmonic-mean AUROC across seen and unseen classes, the area under the risk–coverage curve, and a severe-error rate. A severe error is a confident positive prediction whose finding falls in a different radiologist-defined organ-system group (for example lung parenchyma, pleura, cardiac, or devices) from every true finding. The grouping is fixed before any embeddings are computed, so severity is never judged by the same language model that defines the cost.
Decisive checks. Three results would settle the claim. First, on unseen classes the semantic score from the CNN–ViT pair must beat index disagreement on both risk–coverage and severe-error retention. Second, at matched compute the heterogeneous pair must beat the homogeneous pair when both use the semantic score. Third, the domain-language cost must beat the general-BERT and random costs.
If only the third check succeeds, what remains is an interesting fact about label geometry and not a better uncertainty method. If none succeeds, simple index disagreement suffices for clinical selective prediction, which is itself worth knowing.
References
- Asymmetric Duos: Sidekicks Improve Uncertainty
- Transformer-CNN Cohort: Semi-supervised Semantic Segmentation by the Best of Both Students
- Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs
- Semantic-guided Image Virtual Attribute Learning for Noisy Multi-label Chest X-ray Classification
- Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders
- Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
- Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
- The Benefits of Label-Description Training for Zero-Shot Text Classification
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
- A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology