Repairing Nearest-Neighbor Datastores at New Sites with Training-Only Modalities

Many deployed models see only one cheap sensor, even though richer measurements existed when the training data was collected. A cardiac MRI screener runs on non-contrast cine video because contrast agent is contraindicated for some patients. A night-time pedestrian detector runs on thermal alone. A chest X-ray reader never sees the radiologist's lesion boxes. One approach is to keep the network frozen and adapt it to each new hospital or camera site by retrieval. You store embeddings of that site's labeled archive in an external index, then classify new cases partly by their nearest stored neighbors. Moving to a new site means loading a new index file, with no retraining. This works when distances in the frozen encoder's space reflect true similarity. Under domain shift they often do not. When the cheap modality is weak, two patients with the same scar burden can look far apart in cine space.

The key observation is that the archives at new sites often still contain the rich modality. The archive includes past contrast scans, past paired RGB-thermal recordings, and past boxed X-rays, even though new patients arrive with the cheap modality only. We propose using that site-local privileged data as a label-free teacher for a small offline "reviser." The reviser is a two-layer network that reshapes the frozen unimodal embeddings so that nearest neighbors computed from the cheap modality agree with nearest neighbors computed from the rich one. The reviser runs once over the archive keys and once per incoming query. The backbone stays frozen and the datastore stays swappable. The site needs no new labels beyond those already attached to the archive.

Core Claim

Retrieval-based adaptation fails under shift mainly because the frozen encoder's neighborhoods are wrong at the new site, not because the stored labels are missing. A modality that is available only offline at that site carries enough information to fix those neighborhoods without target-domain fine-tuning. Concretely, let $f$ be the frozen unimodal encoder and $h$ an encoder for the privileged modality. Let $g$ be the reviser, a residual map $g(z) = z + \Delta(z)$. Over the site archive $\{(x_i, x^*_i)\}$, where $x$ is the cheap input and $x^*$ the privileged one, we train $g$ to minimize

$$ \mathcal{L}(g) = \sum_i \mathrm{KL}\big(P^*_i \,\|\, P_i\big) + \beta \sum_i \lVert \Delta(f(x_i)) \rVert^2 . $$

Here $P^*_i$ is the softmax-over-distances distribution of archive case $i$'s neighbors in privileged space $h(x^*)$, and $P_i$ is the same distribution computed in revised unimodal space $g(f(x))$. The second term keeps keys near their original positions, and $\beta$ sets its weight. At test time a cheap-only query $q = g(f(x))$ retrieves labeled archive neighbors. The resulting distribution $p_{\text{kNN}}$ is interpolated with the frozen classifier's prediction as in nearest-neighbor language models: $p = \lambda p_{\text{kNN}} + (1-\lambda) p_{\text{base}}$, with blending weight $\lambda$.

Relation to Prior Work

The retrieval machinery comes from text. Generalization through Memorization: Nearest Neighbor Language Models showed that a frozen LM plus a large datastore can beat a model trained on that data. Nearest Neighbor Machine Translation showed that swapping datastores adapts translation to new domains, with an average gain of 9.2 BLEU over zero-shot transfer. Nearest Neighbor Zero-Shot Inference carried datastore swapping to classification. All three use the encoder's raw geometry, and we keep their interpolation rule.

The direct ancestor of our reviser is Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translation. That paper diagnosed "warped" keys under shift and fixed them with an offline two-layer network trained against a target-domain fine-tuned teacher. It also reported a 0.66 correlation between domain gap and gain. That design needs labeled target parallel data and a fine-tuned teacher, and it is purely textual. We replace the fine-tuned teacher with a modality the site already has, and we revise queries as well as keys, because queries come from a weaker sensor.

Non-Parametric Domain Adaptation for End-to-End Speech Translation is the closest cross-modal precedent. It aligns a text encoder to a frozen speech encoder with an MSE loss, then lets speech queries search a datastore built from text, gaining 12.82 BLEU. Its asymmetry runs the other way from ours: the extra modality builds the datastore, rather than serving as a teacher that corrects retrieval geometry. Its single global MSE alignment also does nothing about warping at a new site.

The privileged-information literature motivates the setting and constrains the design. LGE-Guided Cross-Modality Contrastive Learning for Gadolinium-Free Cardiomyopathy Screening in Cine CMR and Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data pull a cheap encoder toward a rich one with contrastive training. We reuse this to pretrain $f$. Both evaluate in-distribution, however, and Hager et al. flag the 95%-white UK Biobank cohort as a transfer risk. Towards Versatile Pedestrian Detector with Multisensory-Matching and Multispectral Recalling Memory adds a memory of multispectral prototypes. That memory is learned by backpropagation and cannot be swapped per site.

Two works combine privileged information with domain adaptation, but parametrically. Unsupervised Domain Adaptation by Learning Using Privileged Information (DALUPI) proves that overlap is needed only in privileged space, provided the privileged information makes the label independent of the input. That is a strong condition, which our KL-matching objective does not require. Adaptive SVM+ uses privileged features to set per-example slack in a shallow max-margin adaptation. Empirical Risk Minimization for Metric Learning Using Privileged Information supplies the closest idea: privileged distances shaping a learned metric. It works on hand-crafted features in a single domain.

Research Program

The work has three parts.

The first is a diagnostic. Does privileged contrastive pretraining make datastore keys more or less warped at a new site? We pretrain $f$ with and without Hager-style or CC-CMR-style alignment on the source site. We then measure top-1 retrieval label precision of raw unimodal keys on held-out target sites. This tells us whether the reviser is fixing damage that pretraining itself caused.

The second is the reviser and its objective as written above. We vary $\beta$, the neighborhood temperature, and the fraction of archive cases that actually have the privileged modality. That fraction matters because in ChestX-ray8 only a subset carries boxes.

The third is a stress test of the sufficiency assumption. We progressively corrupt the privileged signal and compare how gracefully the revised datastore degrades against DALUPI. For boxes we jitter and shrink them. For attributes we drop the most predictive ones. For RGB we lower its resolution.

Baselines

The naive reference is the frozen privileged-pretrained unimodal model with its own classifier head, applied at the target site with no adaptation. The second reference is the standard symmetric datastore: unrevised unimodal keys from the target archive, queried by unrevised unimodal queries, with $\lambda$ and $k$ tuned on a target validation split. This is the direct analogue of nearest-neighbor translation. Beating it isolates the value of revision.

The strongest parametric competitor is site-level fine-tuning. We continue Hager-style contrastive alignment on the target archive's paired data and then fine-tune the classifier on its labels. Every adaptive method gets the same archive, the same labels, and the same GPU-hour budget for hyperparameter search. Our method's offline reviser training is counted against that budget.

The strongest existing combined method is DALUPI, retrained on source plus the target archive under its published protocol, on the chest X-ray and digit/CelebA tasks it was designed for. The file reports that DALUPI beat DANN, MDD and source-only training on CheXpert/ChestX-ray8 cardiomegaly and even the target oracle. No reported numeric result was found on Lacuna for DALUPI on these tasks, so we will rerun it rather than quote a number.

On the cine CMR cohort, the reference is CC-CMR's reported accuracy of 0.943 LGE-Guided Cross-Modality Contrastive Learning. That figure is in-distribution, and our question is whether a leave-one-vendor-out version keeps it. On KAIST, the memory-based detector reports an 11.39% miss rate in single-modality testing Versatile Pedestrian Detector. No reported result was found on Lacuna for it on CVC-14.

Three ablations isolate the key ingredient. The first is a reviser trained with target labels only, using supervised contrastive neighbors. This is a Cao-style teacher and asks whether privileged structure adds anything beyond labels. The second uses privileged keys directly with unrevised cheap queries, the "asymmetric datastore." The third revises keys only, not queries.

"Beating" a baseline means a higher target-site score than the symmetric datastore and than site fine-tuning at equal budget, with 95% bootstrap intervals that exclude zero. We also require no loss on the source site, which fine-tuning is expected to incur.

Why not just…?

Why not just fine-tune the encoder at each new site? This is the real threat. If the target archive is large, contrastive fine-tuning on it may match or beat the reviser. Our claim is narrower: comparable target accuracy, zero source-site regression, and one frozen backbone with per-site index files. The speech-translation datastore result, which beat in-domain fine-tuning by 3.85 BLEU, suggests retrieval can win outright, but that was text. The fine-tuning baseline at equal budget, swept over archive sizes, will show where any crossover lies. If fine-tuning dominates at every size, the contribution shrinks to deployment convenience, and we will say so.

Why not just put privileged embeddings in the datastore and let cheap queries search them? It sounds like the most direct use of the rich modality. Contrastive alignment, however, pulls each cine embedding toward its own patient's contrast embedding imperfectly. Hager et al.'s image-only gains (AUC 76.35 vs 73.69) show partial, not complete, transfer, so a cheap query lands systematically off the privileged manifold. Cheap-to-cheap neighbors at least share the query's noise. The privileged-keys ablation tests this directly. If it wins, the proposal simplifies to that design.

Why not just learn the retrieval metric from target labels, without the privileged modality? If labels are plentiful this may suffice, and the label-supervised reviser ablation measures exactly that. The argument for privileged structure is that a binary label says nothing about which positive cases resemble each other. Contrast enhancement or lesion location does carry that information, which is the point of ERMML+'s locally adaptive thresholds. We expect the advantage to be largest when target labels are few. We will vary label count to confirm or refute this.

Why not just distill the privileged modality into the student once, as DALUPI or adversarial modality distillation do? Distillation fixes the student's geometry for the source distribution. Adapting it to a new site means retraining. DALUPI's guarantees also rest on the privileged information being sufficient for the label. The sufficiency stress test is designed to show whether neighborhood matching degrades more gracefully than DALUPI as that assumption is violated.

Proposed Experiments

Each testbed provides a real shift between a source and a target site, with the privileged modality present in the target archive but withheld at query time.

The first testbed is CheXpert as source and ChestX-ray8 as target for cardiomegaly and pathology classification. The privileged view is the image cropped to the radiologist box, available on the ChestX-ray8 subset that carries boxes.

The second is KAIST as source and CVC-14 as target, with RGB privileged and thermal-only queries. Retrieval operates on region-proposal embeddings and feeds the proposal classifier.

The third is a leave-one-vendor-out split of the multi-vendor cine CMR cohort, with LGE privileged, if access is granted. The cohort is small (231 subjects), so we report per-vendor results with intervals.

The fourth is a small sanity check on INTERACT, where Adaptive SVM+ reports 83.87% accuracy with an RBF kernel. It lets us compare against a published privileged-adaptation number.

Metrics are AUC or accuracy for classification and log-average miss rate for detection. Alongside them we report top-1 retrieval label precision, query latency, and index size, since the nearest-neighbor translation paper's roughly 100x slowdown and the speech paper's 36 GB datastore make cost a fair concern. Here retrieval happens once per case, not per token.

Four results decide the proposal. First, the revised datastore must beat the symmetric datastore at every target site. Second, it must beat the label-only reviser when target labels are scarce; this is the evidence that the privileged modality, not just extra training, does the work. Third, across sites and synthetic shifts, its gain should grow with measured source–target distance, mirroring the 0.66 correlation in the key-reviser paper. Fourth, it must hold within the fine-tuning baseline's confidence interval at matched budget while leaving source accuracy untouched. Failing the second check would mean the idea reduces to label-supervised metric repair. Failing the fourth would mean the parametric route is simply better for this setting.