A hospital that wants a chest X-ray classifier rarely has thousands of labeled local images. It has a handful, plus a large labeled dataset from some other institution and a frozen foundation model that turns images into feature vectors. Each image can carry several findings at once, and which findings appear together differs from one hospital to the next. This proposal predicts a target image's labels by rebuilding its feature vector as a weighted mix of a few nearby labeled examples, then giving it the same weighted mix of their labels. The method trains no network. Each prediction solves one small linear system. The new ingredient is a penalty that stops the reconstruction from mixing examples whose labels rarely or never occur together.
The underlying technique is locality-constrained coding: a way of representing a point as a combination of nearby "atoms" (reference vectors), so that the representation follows the local shape of the data instead of cutting across empty regions of feature space. We use it here to answer a specific question about label dependencies. Existing multi-label methods impose one global graph of which labels co-occur. Such graphs are skewed by the institution they were estimated at, and they cause models to predict labels that merely tend to accompany the true ones. Our claim is that label dependencies should instead be read locally. The neighbors of an image, and their label sets, already encode which combinations are plausible near that image. Only mutual exclusions need to be imposed explicitly. If this is right, local reconstruction should do two things that independent linear probes and nearest-neighbor voting cannot. It should produce label combinations that are rare in the source data, by mixing single-finding neighbors. It should also stay calibrated when co-occurrence rates shift across domains.
Core Claim
Take a frozen-backbone feature $x$ of a target image, and its $k$ nearest labeled atoms $d_1,\dots,d_k$ drawn from source and few-shot target examples, each with a binary label vector $y_j$. We solve
$$ \min_{c}\ \Big\|x-\sum_j c_j d_j\Big\|^2+\lambda\sum_j c_j^2\,\|x-d_j\|^2+\mu\sum_{j,l} c_j c_l\, y_j^\top N\, y_l \quad \text{s.t. } \sum_j c_j=1, $$
and predict $\hat y=\mathrm{clip}_{[0,1]}\big(\sum_j c_j y_j\big)$. The second term is the standard locality penalty, which makes distant atoms expensive to use. In the third term, $N$ is a nonnegative label-exclusion matrix: entry $N_{ab}$ is large when labels $a$ and $b$ co-occur far less than chance would predict. We estimate it from shrunk pointwise mutual information, keeping only the negative part. With a small ridge term added for stability, the objective is quadratic with one equality constraint, so the code has a closed form per query.
Positive co-occurrence never appears as an explicit prior. It enters only through the label sets of the atoms that geometry selects. This is how the method takes a position on the dispute in the literature. FACT aligns visual primitives with a global comorbidity graph and reports calibration error dropping from 27.51% to 16.22% on ODIR-5K, but notes that the method depends on the prior graph being accurate. ML-VPT finds that leaning on positive co-occurrence produces hallucinated labels, and separates the label space into correlative and discriminative groups. Our design keeps the calibration benefit of structured labels while declining to impose global positive co-occurrence. We do impose exclusions, which are the dependencies least likely to change across institutions.
Relation to Prior Work
The coding machinery comes from classical locality-constrained coding. Four papers make the case for locality over plain sparsity. Efficient Highly Over-Complete Sparse Coding Using a Mixture Model argues that a single global dictionary wastes effort on distant atoms. Linearization to Nonlinear Learning for Visual Tracking shows that local linear reconstruction approximates nonlinear appearance change. Locality-Constrained Concept Factorization keeps bases inside dense regions of the data. Geometrically Regularized Wasserstein Dictionary Learning shows that a distance penalty alone produces sparsity. All four use single-label or unlabeled settings and hand-built features.
The closest prior system is DDLCN. It stacks locality-constrained coding layers over SIFT descriptors, reports near-100% accuracy with one training image per person on Extended YaleB, and outperforms K-SVD and LC-KSVD. It is single-label, uses hand-crafted inputs, and saturates beyond six layers. We drop the depth entirely, swap SIFT for a frozen foundation model, and make the atoms carry label vectors. Label Embedding Based on Multi-Scale Locality Preservation preserves locality only inside the label space. Our method couples feature-space locality to label combinations.
Several papers motivate the regime. Could Giant Pre-trained Image Models Extract Universal Representations? shows that frozen backbones beat full fine-tuning when data is scarce, for example on 1% of COCO. TwinTURBO treats adaptation as retrieving features the frozen model already contains. PRPL does unsupervised domain adaptation over frozen features using pseudo-labels and category-level distribution matching. All three use single-label tasks. PRPL's class-centroid matching is also ill-defined when an image belongs to several classes at once. Two shallow-projection methods carry locality across domains: Locality Preserving Joint Transfer and Locality Preserving Projection for Domain Adaptation. Neither handles label dependencies.
One paper both motivates and constrains the design: Few-shot Adaptation of Medical Vision-Language Models. Its closed-form LP+text method beats both prompt tuning and plain linear probing at one shot, using about 800MB of memory and about 4 seconds of compute. This shows that training-free solvers win at very small sample sizes. Any method we propose must therefore match that cost profile.
Research Program
The atoms are the labeled feature vectors themselves. We do not learn a dictionary. Few-shot target atoms receive a multiplicative weight $w$ inside the locality term, which makes them cheaper to use than source atoms. The only domain alignment is per-domain mean-centering of features. We deliberately keep alignment this weak, so that any gains come from the coding step and not from a stronger alignment step. Three hyperparameters are tuned by leave-one-out on the few target shots: $k$, $\lambda$, $\mu$, and $w$ (four values in total, counting $w$).
The study has three parts. The first is a diagnostic. On frozen features, we test whether images with labels {A, B} are better reconstructed by mixing A-only and B-only atoms than by any single-label atom. We measure the reconstruction residual and the predicted label vector. If frozen features do not compose this way, the central mechanism fails and we will report that.
The second part is the main transfer study across institutions and shot counts. The third part is a set of ablations that separate locality, reconstruction, and the exclusion term from one another.
Baselines
The naive reference point is a frozen backbone with independent per-label logistic regression, trained on source data and applied to the target with no adaptation. A second version adds the few target shots to the training set. This tells us what the frozen features give with no dependency modeling at all.
The strongest training-free competitor is a multi-label version of LP+text from Few-shot Adaptation of Medical Vision-Language Models. It combines per-label visual prototypes with text embeddings and fits learnable class multipliers in closed form, and we run it on the same medical vision-language backbone. Its reported figure is 55.6% one-shot accuracy on NCT-CRC with Quilt-1M, against 52.05% for linear probing and 46.05% for CoOp. That task is single-label, and we found no reported result on Lacuna for LP+text in multi-label cross-institution chest X-ray transfer, so we will rerun it ourselves.
The strongest dependency-modeling competitor is ML-VPT. It reports COCO mAP gains of 1.2% to 8.7% depending on the backbone, but has no reported number under domain shift or at few shots. We run it on the same frozen backbone and the same source-plus-shots data, with the same hyperparameter search budget measured in trials. We also record its memory use and wall-clock time.
A FACT-style global-graph baseline gets the same inputs. It is a graph convolutional network over the source co-occurrence matrix, aligned to frozen features with a small head. FACT reports +2.92% mAP on ODIR-5K within-domain, but no cross-domain number. We also include distance-weighted $k$-nearest-neighbor label voting and global $\ell_1$ sparse-representation classification over the same atoms.
Ablations of our own method remove one ingredient at a time: the exclusion term ($\mu=0$), the locality term (replaced by global $\ell_1$), and reconstruction itself (kNN weights substituted for codes). A further variant replaces the exclusion matrix with a full signed co-occurrence matrix, which tests whether imposing positive co-occurrence hurts under shift.
To count as beating a baseline, the method must improve target mAP and expected calibration error together over LP+text and ML-VPT at 1–8 shots. The improvement must exceed the seed-to-seed standard deviation, measured over at least five draws of the shots. It must hold on at least two of the three transfer pairs, and it must come at LP+text-like compute.
Why not just…?
Why not just use $k$-nearest-neighbor label voting on frozen features? This is the most serious threat. It is local, training-free, and already transfers neighbor label sets. Our method differs in one respect: codes come from reconstruction, so an image lying between an A-only cluster and a B-only cluster gets both labels with calibrated weight. kNN voting can only return combinations its neighbors already carry. The held-out-combination split described below tests this directly. If kNN matches us there, the idea reduces to kNN plus an exclusion filter. We would then report it that way and drop the coding machinery.
Why not just fit linear probes, or LP+text, per label? Independent probes ignore dependencies, and plain probing is weakest at one shot, where Few-shot Adaptation of Medical Vision-Language Models shows a text prior rescues it. A text prior, however, supplies no information about how labels relate to each other. If LP+text matches us on mAP but not on calibration or on rare combinations, then the benefit we claim is specific to modeling dependencies. If it matches us everywhere, our claim fails.
Why not just prompt-tune with label grouping (ML-VPT)? Prompt tuning backpropagates through the backbone. The comparison paper reports up to 28GB of memory for CoOp, and one-shot accuracy below a linear probe. ML-VPT also depends on co-occurrence statistics estimated at the source, which are exactly what shifts across institutions. It may still win at 16 shots. We report the crossover point rather than claim dominance.
Why not just reweight the loss with a source co-occurrence matrix? ML-VPT argues that emphasizing positive co-occurrence produces hallucinated labels. The signed-matrix ablation tests this inside our own method. If including positive co-occurrence does not hurt target calibration, our decision to keep only exclusions is unjustified.
Why not just use global $\ell_1$ sparse coding? DDLCN found that locality beats K-SVD-style coding on SIFT features. Whether that advantage survives on foundation features is untested, and the locality ablation answers it.
Proposed Experiments
We use a frozen general-purpose backbone (DINOv2) and a frozen medical vision-language encoder (MedCLIP). There are three transfer pairs. The first two use NIH ChestX-ray14 as the labeled source, with CheXpert and MIMIC as targets, restricted to their shared findings. The third uses MS-COCO 2014 as source and VOC 2007 as target, restricted to the 20 shared classes; this is a natural-image check on a small label vocabulary, where ML-VPT itself reports reduced gains. For all pairs, target labeled shots are drawn per label at $S\in\{1,2,4,8,16\}$, with five seeds. For long-tail behavior, a within-domain CXR-LT run reports tail-label mAP, a setting where FACT claims strength.
The metrics are mAP, per-class and overall F1, expected calibration error, and a hallucination rate. The hallucination rate is the fraction of predicted positives that are frequent co-occurrence partners of a true label but are absent from the ground truth.
The decisive check is a held-out-combination split. In the target test set, we keep images whose label pair appeared in fewer than five source images. Our method should beat kNN and the per-label probes on these images by a margin larger than the margin on frequent combinations.
A second check concerns shifted co-occurrence. We resample the target set so that one co-occurrence rate is inverted. The global-graph baseline should degrade more than our method.
Each result has a clear interpretation if it goes the other way. If the composition diagnostic fails, frozen features do not compose and the method has no basis. If kNN ties on the held-out split, reconstruction adds nothing over neighbor voting. If $\mu=0$ ties on the hallucination rate, the exclusion term is unnecessary. All three are informative negative results, and we will report them.
References
- FACT: Fuzzy Alignment with Comorbidity Topology for Reliable Multi-Label Medical Image Diagnosis
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning
- Efficient Highly Over-Complete Sparse Coding Using a Mixture Model
- Linearization to Nonlinear Learning for Visual Tracking
- Locality-Constrained Concept Factorization
- Geometrically Regularized Wasserstein Dictionary Learning
- When Dictionary Learning Meets Deep Learning: Deep Dictionary Learning and Coding Network for Image Recognition With Limited Data
- Label Embedding Based on Multi-Scale Locality Preservation
- Could Giant Pre-trained Image Models Extract Universal Representations?
- TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces
- Efficient Pre-Trained Features and Recurrent Pseudo-Labeling in Unsupervised Domain Adaptation
- Locality Preserving Joint Transfer for Domain Adaptation
- Locality Preserving Projection for Domain Adaptation with Multi-Objective Learning
- Few-Shot Adaptation of Medical Vision-Language Models