Disagreement-Gated Duos for Few-Shot Medical Image Classification

A common way to make a large pretrained image model more accurate and better calibrated is to pair it with a small "sidekick" model of a different architecture and average their outputs. A ConvNeXt with a ResNet-34 is one example. Each model carries a fixed weight, and the pair costs only about 10–20% more compute than the large model alone. Those weights are the same for every input. That is a waste whenever which model to trust changes from one image to the next. The waste should be largest in small medical datasets, where a big model fine-tuned on a few dozen images per class tends to be confidently wrong on exactly the images a texture-sensitive convolutional network gets right. We propose to let the weights change per image, using a gate with fewer than ten parameters. The gate reads only label-free signals: how uncertain each model is, and how much the two models disagree. It is fit on out-of-fold predictions, so it never sees the overconfident outputs a model produces on its own training images.

The bet behind this design is specific. In the few-shot regime, a per-instance combination pays off only if it is cheap enough to estimate from tiny data. Disagreement between architecturally different models is the one signal that stays informative when each model's own confidence has been corrupted by overfitting. If the bet holds, a large Vision Transformer adapted with low-rank adapters, plus a small CNN plus an eight-parameter gate, should beat both the static pair and a five-model ensemble on calibration under 20–60 labelled images per class, at roughly the cost of one model. If the bet fails, the experiments are built to show where it fails: either there is no per-instance headroom, or the headroom exists but cannot be estimated from so few labels.

What is new, and relative to what

The closest work is Asymmetric Duos: Sidekicks Improve Uncertainty. It combines a large base model and a cheap, architecturally distinct sidekick by learned weighted averaging of logits with two fixed scalars. The file reports that this matches a 5-model deep ensemble at 4 times less compute. The authors name instance-dependent "Dynamic Duos" as open. That paper supplies our base construction and motivates the gap. We change two things. First, the weights become functions of the input. Second, we move the setting from ImageNet-scale training data to few-shot medical images. In that setting, learning even two scalars on in-sample predictions is already biased, because fine-tuned models are overconfident on the images they were trained on.

A second close relative is Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation. It selects, per instance, the heterogeneous teacher with the lowest entropy, and uses that choice to supervise a compressed student (22.1M parameters versus a 111.8M-parameter teacher pool). Its per-instance rule is pure confidence and is used only during distillation. We use per-instance weighting at test time, and we treat "trust the more confident model" as a baseline to beat rather than the mechanism.

Our gate's key input comes from Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification, which uses disagreement across distinct architectures as an uncertainty signal when single-model self-consistency fails. We use that disagreement as an input for deciding how to combine the two models, not only as an uncertainty score.

The case for instance-conditioning over fixed transformations comes from two lines of adapter work. Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating shows that token-conditioned gating at rank 8 beats static LoRA at rank 32 on LLaMA-3-8B math reasoning (76.02% versus 71.67%). How to Train Your Advisor shows instance-specific advice beating static prompt optimizers, with 85–100% versus 40–60% reward on personalization. Both condition a single model's computation, with large training sets or reinforcement learning. Neither arbitrates between two different models. U-LoRA also pays a 1.3–1.4x latency cost because its weights cannot be merged; our gate adds essentially nothing.

Two papers constrain the design. Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap finds that pools of modern foundation models can be near-redundant, and that naive ensembling can worsen calibration. That is why we measure pairwise redundancy before claiming any gain, and why the gate can push a weight toward zero. Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST finds that foundation models degrade sharply at 20–60 images per class. It also finds that linear probing lags end-to-end fine-tuning. That defines both the regime we target and the two single-model baselines we must include. Finally, Transformer-CNN Cohort shows that pairing a ViT and a CNN prevents shared pseudo-label errors under 1/16 label splits. That is evidence that the two families fail differently when data are scarce. However, that method trains and runs both full networks.

Core Claim

Under few-shot medical adaptation, the best per-instance weighting of a large adapted ViT and a small CNN differs measurably from the best fixed weighting. Most of that difference can be recovered by a gate whose inputs are each model's predictive entropy and the symmetric divergence between the two models' predictions. Concretely, the combined logits are

$$ z(x) = a(x)\, z_L(x) + b(x)\, z_S(x), \qquad [a(x), b(x)] = \mathrm{softplus}\big(\Theta\, \phi(x)\big), $$

where $z_L$ and $z_S$ are the large and small models' logits. The feature vector $\phi(x) = [H_L(x), H_S(x), D(x), 1]$ contains the two predictive entropies $H_L$ and $H_S$, the symmetric KL divergence $D$ between the two softmax outputs, and a constant. $\Theta$ is a $2 \times 4$ matrix, so the gate has eight parameters. When $\Theta$ uses only the constant column, it reduces exactly to the static Duo.

$\Theta$ is fit by minimizing negative log-likelihood on out-of-fold logits. We train both models with K-fold cross-fitting on the few-shot training set, collect each image's logits from the fold model that did not see it, fit $\Theta$ on those, and then retrain both models on all data for deployment. The claim fails in a useful way if the oracle per-instance weighting is close to the static one. In that case dynamic weighting has no room to help, and we report that.

Research Program

The first stage measures headroom before building anything. For each MedMNIST v2 task and each shot count, we compute three numbers on test data: the accuracy and log-loss of the best fixed weighting, the same for an oracle that picks the per-image weight in hindsight, and the redundancy of the pair, measured as the error correlation and the fraction of images where exactly one model is correct. This tells us on which datasets a dynamic gate could matter at all. It also tests whether the diversity-ceiling finding from tabular foundation models carries over to ViT–CNN pairs in medical imaging.

The second stage builds the gate described above and compares two ways of fitting it: out-of-fold logits versus in-sample logits. Our hypothesis is that in-sample fitting learns to over-trust the large model, because its training-set entropy is near zero. Out-of-fold fitting should then matter more as the number of shots shrinks.

The third stage asks whether the gate's low capacity is a feature or a limit. We replace $\phi$ with the concatenated penultimate-layer embeddings of both models and fit a small multilayer perceptron gate. The prediction is that this richer gate wins at full data but loses at 20–60 shots.

The fourth stage checks that heterogeneity, not just a second model, drives the gains. We swap the CNN sidekick for a small ViT of matched compute.

Backbones follow Wu & Papiez. The large model is DINOv2, which the file reports at 92.02% accuracy on DermaMNIST with full-data end-to-end fine-tuning, adapted with LoRA. The sidekick is a ResNet-18 or ResNet-34 trained on the same few-shot data. Compute stays near the Duo budget of 10–20% extra.

Baselines

The naive reference points are single models: DINOv2 with a linear probe, DINOv2 with LoRA, and DINOv2 fine-tuned end-to-end. Each is trained on the same subsampled 20/40/60-shot splits with the same hyperparameter search budget, and each uses post-hoc temperature scaling fit by the same out-of-fold procedure. This keeps calibration comparisons fair. We count it as a win only if the gated Duo improves both expected calibration error and accuracy over the best of these on the majority of MedMNIST 2D tasks.

The strongest existing method is the static Asymmetric Duo of Zhou & Shelhamer: the same two trained models, combined with two learned scalars. We give it the out-of-fold fitting procedure too, so that any gain we report comes from instance dependence rather than from better fitting data. The file reports Duo gains only in aggregate: 10–15% improvement in accuracy and uncertainty metrics over a single model for 10–20% more compute, and parity with a 5-model deep ensemble at 4 times less compute. No reported result found on Lacuna for per-dataset Duo numbers, or for any method on MedMNIST at 20–60 shots. Beating the static Duo means a statistically significant reduction in negative log-likelihood and expected calibration error, paired across at least five resampled few-shot splits, with no loss in accuracy.

The compute-heavy references are a homogeneous deep ensemble of five DINOv2-LoRA runs with different seeds, and a full heterogeneous pair of DINOv2 with EfficientNet-B4, the best CNN in the file at 88.13% on DermaMNIST with full data. Matching the five-model ensemble's calibration at about a fifth of its compute would count as success. For context only, the file reports full-data MedMNIST reference points of 0.927 average 2D AUC for Google AutoML Vision and 0.863 AUC for ResNet-50 3D, from MedMNIST v2. These are not few-shot numbers.

Four ablations isolate the key ingredients. A confidence-only gate keeps the two entropies but drops the disagreement term, which isolates the role of cross-model disagreement. In-sample fitting versus out-of-fold fitting isolates the estimation procedure. The embedding MLP gate isolates gate capacity. The homogeneous small-ViT sidekick isolates architectural diversity.

Why not just…?

Why not just keep the static Duo weights? This is the real threat. If per-instance headroom is small, eight parameters will add estimation noise and nothing else. The first-stage oracle measurement settles it directly. If the oracle-versus-static gap is negligible on most tasks, we will report that the few-shot medical regime does not need Dynamic Duos. That result would itself answer the open question posed by Zhou & Shelhamer.

Why not just trust whichever model is more confident on each image, as in adaptive knowledge amalgamation? The per-instance teacher selection in Shen et al. assumes confidence is informative. Few-shot fine-tuning is exactly the condition in which a large model's confidence stops being informative. The confidence-only gate and a hard argmin-entropy selector are both in our baseline set. Our claim requires the disagreement term to add measurable value over them, especially at 20 shots.

Why not just fine-tune the large model end-to-end, or average five copies of it? Wu & Papiez report that end-to-end fine-tuning beats linear probing but still degrades sharply at 20–60 images per class. Homogeneous copies share the same inductive bias, which Transformer-CNN Cohort shows leads to correlated errors under label scarcity. Both are run as baselines under matched tuning budgets. If the five-copy ensemble matches us on calibration, our remaining claim is cost alone, and we will say so.

Why not feed the sidekick's features into the large model's adapter, in the style of U-LoRA gating, instead of combining logits? We considered making this the core of the proposal and rejected it. It adds many parameters that must be fit from tens of images. It makes the adapter unmergeable, inheriting U-LoRA's 1.3–1.4x latency. And it forces the two models to run in sequence. The embedding-MLP gate tests a milder form of the same idea: whether richer conditioning helps at all with few labels. If it does, conditioning the adapter on sidekick features becomes the natural follow-up.

Decisive Checks

We use all twelve MedMNIST v2 2D tasks, subsampled to 20, 40, and 60 images per class with five random splits each, plus the 3D tasks at whatever shot counts their sizes allow. Metrics are accuracy, AUC, negative log-likelihood, expected calibration error, selective-classification area under the risk–coverage curve, and measured FLOPs.

Three outcomes decide the proposal. The first is whether oracle per-instance weighting has meaningful headroom over static weighting in the few-shot regime. The second is whether the eight-parameter gate fit out-of-fold captures a substantial share of that headroom while the in-sample-fit gate and the embedding MLP gate do not. That pattern would show the gain comes from careful low-capacity estimation rather than capacity. The third is whether the gains shrink when the CNN sidekick is replaced by a small ViT. That would show architectural disagreement, not model count, is what the gate exploits.

A positive result on all three would give a practical recipe for calibrated small-cohort medical classifiers at near single-model cost. A negative result on the first would close the "Dynamic Duos" question for this regime.