When a model combines several views of the same case, such as frontal and lateral chest X-rays or several camera angles of a product, one bad view can drag the combined prediction below what the best single view would give on its own. Current fixes judge each view as a whole: a view is trusted or down-weighted with a single number. Real corruptions are rarely that uniform. Sensor grain or motion blur ruins fine texture while leaving the overall anatomy readable. An occluded or uninformative camera angle keeps sharp edges but loses the object's identity. A frozen pre-trained network separates these cases for free, because its shallow layers respond to texture and its deep layers respond to semantics. We propose to score the reliability of every view at every depth of a frozen backbone, separately for each test case and without using labels, and to combine many tiny per-(view, depth) classifiers according to those scores.
The setting that makes this matter is the small-sample regime common in clinical work, often fewer than 100 labeled patients (Bria et al.). With so few labels, a learned cross-view attention head or an evolutionary architecture search cannot be fit without memorizing noise. Our reliability scores are therefore computed from feature statistics alone, and the only supervised parameters are low-dimensional linear probes. Because the score of a view at a given depth is measured against that view's own clean-feature distribution, not against other views, the method still works with only two views. Two-view pairs are exactly where pairwise self-weighting schemes collapse.
Core Claim
Our claim is that, in frozen features, corruption is localized by depth, and that gating at the level of (view, depth) recovers signal that per-view gating throws away. Concretely: under corruptions that damage one frequency band of one view, a reliability-weighted ensemble of per-(view, depth) linear probes should beat the same ensemble with one reliability weight per view. It should do so with 16 to 128 labeled cases, and it should match per-view gating when corruption is absent or affects all depths.
Two outcomes would refute the idea. The first is a negative diagnostic: corruption shifts all depths of the frozen backbone about equally. The second is a negative comparison: collapsing the depth axis costs nothing. Either result would be informative, and the program below tests both directly.
Relation to Prior Work
The motivation comes from the robust multi-view literature. MVCAN named the "noisy-view drawback," in which adding a noisy view hurts clustering. It handles the problem with separate encoders per view and weights based on mutual information. SEM weights pairs of views in contrastive learning by how much they disagree, and it notes that this weighting degenerates when only two views exist. RINCE bounds the gradients that noisy positive pairs contribute to contrastive pre-training. TEF uses evidential uncertainty and architecture search to build robust fused views.
All four assign one weight or one uncertainty per view, and all four train substantial parameters. We differ in three ways: we weight at the granularity of (view, depth), we weight per test case at inference time, and we spend no labels on the weighting. MVP-N supplies real uninformative views together with human informativeness labels, and we use those labels to validate our weights. UGDA tackles few-shot multi-view learning with missing views, but it works on flat, pre-extracted vectors. Ours is the corrupted-but-present case, with features taken from inside a foundation backbone.
The method itself borrows from the hierarchical-fusion and frozen-feature lines. Hwang and Liu showed that concatenating Conv1 through Conv5 of a frozen network beats any single layer for detecting boundaries (ODS 0.741 against 0.699 for the best single layer, Conv2). Dorster et al. found that frozen DINOv2 patch embeddings outperform the global summary token, which overfit. Yu et al. report that intermediate vision-transformer layers are more robust to distribution shift than final layers.
That last finding appears to contradict the view that "shallow layers amplify noise" (Lee). We read the two as complementary: which depth is robust depends on the corruption type. That dependence is precisely why a fixed choice of layers, or a single concatenation, is the wrong design.
Two papers constrain the design. MFM-Geom shows that structured heads on frozen embeddings help in low-data regimes, which supports our choice to keep the head tightly structured. The curse-of-dimensionality concern rules out concatenating every layer of every view into one vector when $N \ll D$.
Research Program
We use two frozen backbones. The first is DINOv2 ViT-B/14, following Dorster et al. We tap blocks 3, 6, 9 and 12; in a transformer these differ in abstraction, not in spatial resolution. The second is a ConvNeXt of the kind CheXFusion uses (CheXFusion), tapped at its four stages, which form a true resolution pyramid. For each view $v$ and depth $l$ we mean-pool the patch features and project them onto the top $k \approx 32$ principal components, fitted on unlabeled images from the target domain. On these components we fit a ridge-regularized logistic probe that produces class probabilities $p_{v,l}(x)$.
To score reliability, we take the feature $z_{v,l}(x)$ and compute a typicality distance $d_{v,l}(x)$. This is a Mahalanobis distance with Ledoit–Wolf shrinkage, measured against the clean unlabeled feature distribution for that view and depth, and divided by its 95th percentile on held-out clean data so that distances are comparable across depths. The fused prediction is
$$ \hat p(x) = \sum_{v,l} w_{v,l}(x)\, p_{v,l}(x), \qquad w_{v,l}(x) = \frac{\pi_{v,l}\, e^{-\tau d_{v,l}(x)}}{\sum_{v',l'} \pi_{v',l'}\, e^{-\tau d_{v',l'}(x)}}, $$
where $\pi_{v,l}$ is a prior weight for each (view, depth) pair, set from that probe's cross-validated log-likelihood on the training labels, and $\tau$ is a single temperature tuned by cross-validation. Beyond the probes, the supervised part of the model has one scalar plus $V \times L$ priors. This is why it remains viable at $N=16$.
The research program has three parts. First, a diagnostic study. We apply controlled corruptions to one view: Gaussian noise, JPEG artifacts, defocus blur, and occlusion of half the image. We then measure how much $d_{v,l}$ shifts at each depth. The prediction is that noise and JPEG shift shallow depths most, while occlusion shifts deep depths most. Second, the method and its ablations. Third, a check of the weights against human judgments of how informative each view is.
Baselines
The naive reference is the best single view with a final-layer probe, using the same backbone, the same PCA, and the same ridge tuning. We include it because the noisy-view drawback is defined relative to this baseline. CheXFusion reports that a single view reaches mAP 0.340 on MIMIC-CXR-LT, against 0.357 for concatenation and 0.372 for transformer fusion, all at full data. Our method must at least match this baseline under corruption and beat it when views are clean.
The second reference is concatenation: all views and all depths are concatenated, reduced to the same total dimension with PCA, and fed to one ridge probe. This is the Hwang-and-Liu recipe extended across views. It has the same parameter budget as our method and the same cross-validation grid.
The key ablation, and in our view the strongest competitor, is per-view gating. It is identical to our method except that $d_v(x)$ is computed on the final layer and one weight is shared across all depths of a view. It is the label-free, test-time analogue of MVCAN- and SEM-style weighting, and it isolates the depth axis exactly. We count our method as beating it only if the gain on scale-localized corruptions is at least twice the standard deviation across 20 resampled training sets, with no significant loss on clean data.
We also run SEM and RINCE as published, using pre-training on the same unlabeled pool followed by the same probes. This tests whether learned robust objectives beat label-free gating in the low-label regime. Their flat-benchmark numbers do not transfer to our setting, and no reported result found on Lacuna for either method on PadChest or MVP-N in the few-shot regime.
Finally, we compare to end-to-end fine-tuning. On PadChest, the strongest reported multi-view number is REMEDIS at AUC 69.0% with 1% of the data and 73.7% with 10% (Pre-training benchmark). We will report against these figures at the same 1% and 10% fractions, matching their multi-label AUC protocol. Because REMEDIS weights may not be available, we also fine-tune DINOv2 end to end ourselves, on the same splits and with the same hyperparameter-search budget measured in GPU-hours. On MVP-N, the reported reference is a deep-attention network at 91.61% multi-view accuracy with the full data (MVP-N). no reported result found on Lacuna for MVP-N with 16 to 128 training sets.
Why not just…?
Why not just gate each whole view? This is the real threat, and our main experiment exists to settle it. If corruptions in frozen features spread across all depths, per-view gating loses nothing, and the diagnostic study will show that before any comparison is run. We expect per-view gating to win or tie on occlusion and uninformative angles, where the damage is semantic, and to lose on noise and JPEG artifacts, where a view's deep features are still usable but final-layer typicality alone is a weak signal. If the gap does not appear, we will report a negative result rather than add machinery to rescue it.
Why not just train the probe with corruption augmentation? Training on blurred and noisy copies of the training images is cheap and could teach the probe to ignore damaged features. It is a serious alternative, and we will run it both on its own and combined with our gating. Its weakness is that it covers only the corruptions it was trained on. To test that, we hold out corruption families: train with noise, test with JPEG or occlusion. Our gating requires no assumption about which corruptions to expect.
Why not just use the final layer, or fine-tune end to end? The final layer discards local structure. The evidence is the 0.741 versus 0.699 ODS result of Hwang and Liu and the [CLS]-token overfitting reported by Dorster et al. Our depth ablation tests this claim directly. Fine-tuning gave large gains at 1% of PadChest in the pre-training benchmark, but that fraction is roughly 1,600 images. The fine-tuning comparison at N≤128 described in the Baselines section decides which side of the tension described in the research file holds in the regime we care about.
Proposed Experiments and Decisive Checks
We use PadChest, with frontal and lateral views as a genuine two-view problem. We evaluate at N ∈ {16, 32, 64, 128} patients and at the 1% and 10% fractions. In each test case, one view, chosen at random, is corrupted at three severities. We also use MVP-N with its real uninformative views, at 16 to 128 training sets, and the MIMIC-CXR-LT split to check transfer to long-tailed labels. Metrics are macro AUC on PadChest, mAP on MIMIC-CXR-LT, and accuracy and MCDU on MVP-N. MCDU is MVP-N's measure of how confident a model is on uninformative views.
Three checks decide the outcome. The first is whether the depth profile of $d_{v,l}$ separates corruption types. The second is whether (view, depth) gating beats per-view gating on noise and JPEG at N≤64 while tying on clean data. The third is whether, on MVP-N, the per-view weights $\sum_l w_{v,l}$ have a Spearman correlation above 0.4 with the human informativeness labels, compared with the correlation achieved by final-layer typicality alone.
References
- Towards AI-driven Next Generation Personalized Healthcare and Well-being
- Investigating and Mitigating the Side Effects of Noisy Views for Self-Supervised Clustering Algorithms in Practical Multi-View Scenarios
- Self-Weighted Contrastive Learning among Multiple Views for Mitigating Representation Degeneration
- Robust Contrastive Learning against Noisy Views
- Trusted Multi-View Classification via Evolutionary Multi-View Fusion
- MVP-N: A Dataset and Benchmark for Real-World Multi-View Object Classification
- Few-Shot Partial Multi-View Learning
- Pixel-wise Deep Learning for Contour Detection
- Towards Early Detection: AI-Based Five-Year Forecasting of Breast Cancer Risk Using Digital Breast Tomosynthesis Imaging
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment
- Multiple Object Tracking via Feature Pyramid Siamese Networks
- A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification
- CheXFusion: Effective Fusion of Multi-View Features using Transformers for Long-Tailed Chest X-Ray Classification
- A Comprehensive Benchmark of Supervised and Self-supervised Pre-training on Multi-view Chest X-ray Classification