A model trained at one hospital often loses much of its accuracy at another. On the Camelyon17 pathology task, standard training drops from about 93% accuracy on held-out patches from training hospitals to about 70% at an unseen hospital (WILDS). The deeper problem is that nobody knows how large the drop will be before collecting labels at the new site. That same benchmark lists label-free model selection for new domains as an open problem. Multi-view learning offers an old answer that has not been tested in this setting. When each example comes with two separate descriptions ("views"), such as the frontal and lateral X-ray of the same chest, and the two views are conditionally independent given the label, meaning that once you know the diagnosis, one view tells you nothing more about the other, then how often two single-view classifiers disagree on unlabeled data bounds how often they are wrong (PAC Generalization Bounds for Co-training). Disagreement can be computed at a new hospital without any labels.
This proposal claims that raw disagreement is a biased gauge of error, and that the bias has a single named source that can be measured. The source is the rate at which both views are wrong in the same way, which is exactly what happens when both views pick up a shared acquisition artifact. We propose to estimate this "correlated-error mass" on labeled source hospitals, carry it to the new hospital as a correction, and test directly whether the correction transfers across institutions. If it transfers, cross-view disagreement becomes a usable label-free estimate of accuracy at a new site. If it does not, we learn that view dependence itself shifts between hospitals, which would be a useful negative result for both fields.
Core Claim
Take two classifiers $h_1, h_2$, one per view, with error rates $\varepsilon_1, \varepsilon_2$ on some target distribution. Every example falls into one of four cases: both right, exactly one right, both wrong with the same answer, or both wrong with different answers. Counting disagreements across these cases gives the identity
$$\varepsilon_1 + \varepsilon_2 = d + P(\text{both wrong}) + a,$$
where $d = P(h_1 \neq h_2)$ is the disagreement rate, which is observable without labels, and $a = P(h_1 = h_2 \neq Y)$ is the correlated-error mass: the probability that both views agree on a wrong answer. Under conditional independence, both unobserved terms are products of small per-view error rates, so $d$ nearly determines the error. When the views share a shortcut, such as patient positioning, portable-scanner markers or exposure settings that affect frontal and lateral films together, $a$ grows. Disagreement then understates error, and it does so most at the hospitals where the shortcut breaks.
Our estimator models the unobserved terms as a dependence coefficient. We define $\kappa$ as the ratio of the observed both-wrong mass to the value it would take under independence, fit $\kappa$ on labeled source sites (per pathology, and optionally as a function of simple image metadata), and solve the identity for target error using target disagreement. The question that decides whether this works is whether $\kappa$ is stable across institutions. Stability is an empirical claim, and it can fail.
Relation to Prior Work
Theory supplies the starting point but assumes away the hard part. PAC Generalization Bounds for Co-training proves that disagreement bounds error under strict independence. Multi-View Active Learning in the Non-Realizable Case relaxes this to an expansion condition, and Efficient Co-Training of Linear Separators under Weak Dependence and Fast Co-Training under Weak Dependence via Stream-Based Active Learning relax it further to weak dependence. All four use agreement to train on one distribution. None uses it to audit a model on a shifted one, where the dependence may itself have changed.
Empirical work motivates the worry that dependence breaks the bound in practice. Watch, Listen & Learn observed that correlated views make the same errors and reinforce them during co-training. Weakly Supervised Natural Language Learning Without Redundant Views showed co-training losing to self-training when views are not truly separate (MUC-6 F-measure 60.1 versus 64.1). Beyond Independence and Weakly-Supervised Multimodal Learning on MIMIC-CXR both show that fusing views as if independent overcounts shared evidence. The second works on the same frontal/lateral pairs we use, but only inside MIMIC-CXR, and it learns representations rather than estimating error.
The closest cross-dataset work does something different in each case. CoMet uses meta-features from earlier datasets to pick pseudo-labels during tabular co-training on a new dataset. It never measures view dependence or tests shifted targets. Maximizing conditional independence for unsupervised domain adaptation enforces independence between features and domain, not between views, and it needs target data during training. Multi-view Domain Generalization for Visual Recognition uses RGB and depth views to regularize training. None of these produces a label-free accuracy estimate for an unseen site.
Research Program
Testbed. The main testbed is paired frontal and lateral chest radiographs from three institutions: MIMIC-CXR, CheXpert (CheXpert) and PadChest. These are natural views that a clinician would regard as partly redundant and partly complementary. For each pathology, we treat detection at a fixed operating threshold as a binary task, so that error rates are well defined. We also report AUROC.
Model pool. We train a pool of roughly one hundred per-view classifier pairs by varying architecture, augmentation, training length and regularization. This spread produces a wide range of target accuracies to rank.
Protocol. We use leave-one-institution-out. We fit $\kappa$ on two institutions, then predict error on the third from its unlabeled pairs, comparing the prediction against the true labeled error only at the end. Within each institution, acquisition strata, such as portable versus fixed scanners and patient-position groups, serve as additional smaller domains. These let us see how $\kappa$ varies with acquisition.
Controlled companion experiment. Chest X-rays do not let us set the degree of dependence. We therefore build a synthetic two-view version of the modified CIFAR construction from Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup. A spurious patch is added to both views with a tunable strength and is removed or reassigned at test time. This lets us sweep $a$ from zero to large, check the identity exactly, and find the dependence level beyond which raw disagreement becomes misleading.
Baselines
The naive reference is the practice WILDS criticizes: rank models by in-distribution validation accuracy, and predict target error as source error. It uses the same pool and source-site labels as our method. We beat it if our target-error predictions have lower mean absolute error and our checkpoint rankings have higher Kendall rank correlation with true target accuracy.
The second baseline is a confidence proxy: the average maximum predicted probability of the fused frontal-plus-lateral model, after temperature scaling on source validation data. This is the most common label-free estimate. It gets the same calibration data we use to fit $\kappa$. The file reports no numbers for it on these datasets, so we will produce them ourselves.
The third baseline is the key ablation: raw cross-view disagreement with no correction, that is, $\kappa$ fixed at independence. It isolates whether the dependence correction, rather than the use of two views, drives any gain. The prediction is that raw disagreement ranks models reasonably but consistently underestimates error, and does so most at the institution whose acquisition differs most.
The fourth baseline is disagreement between two differently seeded models on the same view. This is the most serious competitor, discussed below. It is run with identical compute, one extra model per pair.
The ceiling is oracle selection using target labels. We have no reported result on Lacuna for label-free accuracy estimation or model selection on MIMIC-CXR, CheXpert or PadChest. As published reference points we use the CheXpert DenseNet121 AUROCs of 0.97 for pleural effusion and 0.94 for edema (CheXpert), as a sanity check that our single-view models are competitive. We use the Camelyon17 figures of about 93% versus about 70% (WILDS) as the size of shift a useful estimator must track. Counting as a win means beating the confidence proxy and same-view disagreement on both mean absolute error and selection regret, meaning the drop in target AUROC relative to the oracle choice, in at least two of the three held-out institutions.
Why not just…?
Why not just select models on in-distribution validation data? WILDS shows that in-distribution accuracy can sit about 23 points above out-of-distribution accuracy, and it names in-distribution selection as unreliable. Our naive baseline measures how badly it ranks the pool on held-out hospitals. If it ranks as well as our estimator, the proposal adds nothing for selection. It could still add something for predicting the size of the drop.
Why not just trust the model's own confidence? Confidence is calibrated on the source data and inherits the source's shortcuts. A model that relies on a scanner marker is confidently wrong where the marker's meaning changes. The confidence baseline tests this directly. We expect it to degrade on the institution with the largest acquisition shift. If it does not, cross-view structure is unnecessary.
Why not just use disagreement between two seeds on one view? This is a real threat. It needs no paired data and may already track error. Its weakness is structural: two seeds trained on the same view share the same shortcuts, so their agreed-upon errors should be larger than those between frontal and lateral models. Our identity predicts exactly that, and we will measure the correlated-error mass for both kinds of pair on labeled target data after the fact. If same-view pairs show no more correlated error than cross-view pairs, the physical-view argument fails. In that case we would report the correction as a general improvement on disagreement-based estimation and drop the specifically multi-view claim.
Why not just remove the dependence during training instead of correcting for it? Methods that decorrelate views or remove domain signal (Beyond Independence, Maximizing conditional independence for unsupervised domain adaptation) change the model being audited, and invariance methods mostly failed to beat standard training in WILDS. Our estimator is applied after training, to any model pool. The two approaches can be combined: we include models trained with such regularizers in the pool and check whether they show smaller $\kappa$.
Decisive Checks
Three results settle the proposal. First, in the synthetic sweep, the estimation error of raw disagreement should grow roughly in proportion to the injected correlated-error mass $a$, as the identity dictates, while the corrected estimator stays flat until the test-time shortcut reassignment changes $a$ itself. Second, on the chest X-ray data, the fitted $\kappa$ must vary across institutions by less than it varies across pathologies. If it instead varies more across institutions, conditional dependence is itself site-specific, the correction cannot transfer, and we will report that as the main finding. Third, under leave-one-institution-out, the corrected estimator must beat same-view seed disagreement on selection regret. This is the comparison on which the value of genuinely distinct views rests.
References
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- PAC Generalization Bounds for Co-training
- Multi-View Active Learning in the Non-Realizable Case
- Efficient Co-Training of Linear Separators under Weak Dependence
- Fast Co-Training under Weak Dependence via Stream-Based Active Learning
- Watch, Listen & Learn: Co-training on Captioned Images and Videos
- Weakly Supervised Natural Language Learning Without Redundant Views
- Beyond Independence: Learning Correlated Views for Variational Incomplete Multi-View Clustering
- Weakly-Supervised Multimodal Learning on MIMIC-CXR
- CoMet: A Meta Learning-Based Approach for Cross-Dataset Labeling Using Co-Training
- Maximizing Conditional Independence for Unsupervised Domain Adaptation
- Multi-view Domain Generalization for Visual Recognition
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
- Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup