When a model trained on plentiful labeled data from one domain (say, product photos) must work in another (say, clip art) given only one or a handful of labeled target images per class, two things go wrong at once. The target classifier is badly underdetermined: a few points in a 768-dimensional feature space cannot pin down a decision boundary. The obvious fix is to borrow structure from the source, but that also fails. The directions that separate classes best in the source domain often include source-specific cues, such as textures and rendering styles, that mean nothing in the target. This note proposes a single low-dimensional projection, learned from labels, that keeps only the class-separating directions the source data and the few target shots agree on. Directions where the source shows class structure but the target shots do not, or where they disagree, are treated as noise. A linear classifier is then trained jointly with the projection, on source and target examples together, inside that small subspace.
The opportunity is that this agreement can be measured even with one target example per class, because it compares class means across domains rather than estimating a target distribution. In its first form the projection is a closed-form generalized eigenproblem on frozen backbone features. It costs seconds, needs no unlabeled target pool, and needs no multi-domain deep generative model. If it works, practitioners get a cheap and auditable alternative to meta-learned data mixing and to backbone tuning. They also get a direct test of a claim the literature asserts but has not isolated under extreme scarcity: that the main failure is borrowing the wrong source directions, not having too few parameters.
Core Claim
In few-shot supervised domain transfer with frozen features, the subspace the classifier may use matters more than the classifier itself. That subspace should be chosen by a discriminant criterion with two parts. Its "signal" is the cross-domain agreement of class-mean geometry. Its "noise" adds the per-class source-to-target mean shift to ordinary within-class scatter. Restricting a jointly trained margin classifier to this subspace should beat two alternatives at 1 to 5 shots. The first is source-discriminant subspaces, which fall into what the multi-source literature calls the texture trap. The second is full-dimensional probes regularized toward the source classifier.
The nearest prior work frames what is new. Unsupervised Visual Domain Adaptation Using Subspace Alignment aligns per-domain PCA bases in closed form. It ignores labels, and it needs enough target data for a target covariance, which does not exist at one shot. Kernelized Bayesian Transfer Learning is the closest relative: it jointly learns domain-specific kernel projections and a shared classifier with scarce target labels. But it fits a separate projection per domain under variational inference, and it has no term that explicitly penalizes directions whose class arrangement differs between domains. We use one shared projection and make cross-domain disagreement an explicit cost.
Subspace Identification for Multi-Source Domain Adaptation supplies the motivating diagnosis. It separates label-relevant features that are domain-specific (the texture trap) from those that are domain-invariant. Its identifiability argument, however, requires $n+1$ source domains and a deep variational autoencoder trained with unlabeled target data. We target the opposite regime: one or two sources, a few labeled target shots, and a linear operator. Progressive Mix-Up for Few-Shot Supervised Multi-Source Domain Transfer works in exactly our data regime (1-shot target, no unlabeled target data). It mixes pixels and features in high dimension with a target-to-source curriculum and meta-learning, and it builds no subspace. We take its empirical lesson, that abundant source data can wash out target signal, as a design constraint rather than a curriculum heuristic.
Method
Let $x \in \mathbb{R}^D$ be frozen backbone features. For class $c$ among $K$ classes, let $\mu^S_c$ be the source class mean and $\mu^T_c$ the target class mean computed from the few target shots, each centered by its own domain's grand mean. Let $\Sigma^S_W$ be the shrunk within-class covariance of the source. We define an agreement matrix and a nuisance matrix:
$$A = \tfrac{1}{2}\sum_{c=1}^{K}\left(\mu^S_c \mu^{T\top}_c + \mu^T_c \mu^{S\top}_c\right), \qquad N = \Sigma^S_W + \beta \sum_{c=1}^{K} (\mu^S_c - \mu^T_c)(\mu^S_c - \mu^T_c)^\top + \epsilon I,$$
where $\beta \ge 0$ weights the domain-shift scatter and $\epsilon$ is a small ridge. A direction $w$ scores high on $w^\top A w$ only if source and target spread the classes in the same way along it. It scores zero if one domain shows class structure there and the other does not, and negative if they disagree. Because target-shot noise is uncorrelated with source means, it partly averages out across classes. This is why the criterion stays usable at one shot when $K$ is moderate. The initial projection $W_0 \in \mathbb{R}^{D \times d}$ is given by the top $d$ generalized eigenvectors of
$$A w = \lambda N w,$$
with $d$ chosen as the number of eigenvalues above a threshold derived from a permutation null, obtained by shuffling target-shot class labels. This removes the hand-tuned dimension that both Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning and Subspace Alignment leave to trial and error.
The second stage is the task-informed part. Starting from $W_0$, we alternate two steps in the manner of Merging SVMs with Linear Discriminant Analysis: A Combined Model, which fixes the projection to fit margin hyperplanes and then refits the projection on the support vectors. Here the classifier is a multi-class hinge or logistic model trained on source and target examples, with target examples up-weighted. The projection update keeps $W$ within a ball around the span of $W_0$, plus the $A$-versus-$N$ criterion as a regularizer. That regularizer is the guard against the failure the research file flags for task-driven reduction under scarce labels: task gradients overfitting the projection to idiosyncratic training points. Learning the Optimal Discriminant SVM With Feature Extraction supplies the solver machinery and the evidence that joint projection-plus-margin training beats sequential pipelines in a single domain. We extend it to two domains with a cross-domain regularizer. The target weighting addresses the tension the file identifies between source-initialized subspace methods and P-Mixup's finding that target-first ordering matters. In our method the subspace is defined by target agreement from the outset, and the source only enters through directions the target confirms.
Research Program
The work has three parts. The first is building and characterizing the closed-form operator. That includes behavior as shots go from 1 to 16, as the number of classes varies (the averaging argument predicts degradation at small $K$), and as $\beta$ varies. The second is the joint margin refinement, with the explicit question of whether it adds anything beyond $W_0$ at 1 shot or only at 5 or more. The third is a mechanistic check on whether the method removes texture-trap directions. We will construct controlled source data by planting a synthetic class-correlated feature that exists only in the source, for example by appending class-dependent style statistics. We will then measure how much the learned subspace overlaps with it compared with source-only discriminant analysis.
A secondary extension tests vision-language backbones. There, the text embedding of each class name can stand in for $\mu^S_c$, which turns the method into a text-agreement subspace. Few-shot Adaptation of Medical Vision-Language Models shows that text-anchored linear probes are strong in this regime, so this extension is a demanding test.
Baselines
The naive reference point is a linear probe on frozen features, trained only on the target shots, plus the same probe trained on pooled source and target data. These tell us whether source data helps at all and whether naive pooling hurts. Few-shot Adaptation of Medical Vision-Language Models reports that a plain linear probe reaches 52.05% at 1 shot on NCT-CRC histology, which calibrates how weak this reference is.
The strongest existing method in our exact data regime is P-Mixup (Progressive Mix-Up for Few-Shot Supervised Multi-Source Domain Transfer). It reports 74.99% mAP on Office-Home and 75.50% mAP on the four-domain DomainNet subset in its 10-class, 1-shot protocol. No code for it appears in the file, so we will reproduce it under its own protocol to confirm those numbers. We will then also run it on our frozen backbone with the same source and target splits, the same feature extractor, and a hyperparameter search of equal trial count. That separates its gains from its backbone tuning. Beating it means matching or exceeding its reproduced mAP under the shared frozen backbone, at a fraction of its compute. Matching its originally reported number while P-Mixup trains end to end would be a strong secondary result, not a requirement.
For the medical extension, the strongest competitor is LP+text, which reports 55.6% at 1 shot on NCT-CRC. Beating it means exceeding that figure with the same QuiltNet backbone and seeds. In the cross-domain few-shot setting, DIPA (Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning) is the strongest parameter-efficient tuner. The file reports only relative gains for it (up to 7.7% seen and 5.3% unseen on Meta-Dataset) and no absolute accuracy, so we will rerun it on our splits. Multi-source Few-shot Domain Adaptation (Multi-source Few-shot Domain Adaptation) reports only mean improvements, and it uses unlabeled target data that our setting forbids. It is excluded from the headline comparison and discussed separately.
Three ablations isolate the key ingredient. The first, $\beta = 0$, drops the shift scatter. The second replaces $A$ with the pooled between-class scatter, which gives plain supervised discriminant analysis on pooled data and tests whether cross-domain agreement, rather than supervision alone, is what matters. The third uses source-only discriminant analysis. A fourth ablation drops the joint margin refinement, which tests the "task-informed" claim separately from the subspace choice.
Why not just…?
Why not just fine-tune the source classifier on the target shots with a penalty pulling weights back toward the source solution? This is the most serious threat. It is cheap, uses source knowledge, and stays full-dimensional. Our method differs in one respect: it lets the source contribute only along directions the target confirms, whereas a weight-space penalty imports source-specific directions wholesale. We will run this baseline with the penalty strength tuned on the same validation shots. If it matches our method across Office-Home pairs with large style gaps, such as Product to Clipart, the subspace claim fails and we will report that. The planted-texture experiment is where the two should diverge most clearly.
Why not just apply PCA or Subspace Alignment and then a classifier? Subspace Alignment needs a target covariance, which one shot cannot provide, and its bases ignore labels entirely. In a single domain, joint margin methods already beat sequential PCA-then-SVM pipelines: Merging SVMs with Linear Discriminant Analysis: A Combined Model reports 97.5% versus 94.7% for PCA on XM2VTS. We include SA, with the source PCA standing in where target PCA is degenerate, so that this gap can be measured across domains rather than assumed.
Why not just use ordinary supervised discriminant analysis on pooled source and target data? This is the cleanest test of the core claim. Pooled discriminant analysis is dominated by source data and should recover source-specific directions. If it ties with our method, then agreement weighting adds nothing. It appears both as an ablation and as a full baseline.
Why not just tune prompts or adapters? In the medical regime, CoOp reached only 46.05% at 1 shot on NCT-CRC while using 28GB of memory (Few-shot Adaptation of Medical Vision-Language Models). DIPA is the stronger adapter-style competitor and is run as a baseline. Our method could also sit on top of DIPA-adapted features, which we test as a combination.
Proposed Experiments
The main benchmarks are Office-Home and the Clipart/Painting/Real/Sketch subset of DomainNet. We use P-Mixup's 1-shot target protocol with no unlabeled target data, and also 3- and 5-shot variants, and run both single-source and multi-source configurations. For multi-source, $A$ is computed for each source-target pair and the pairs are summed. We report accuracy and mAP over at least ten seeds of shot sampling, along with wall-clock time and memory.
PACS serves as a small-$K$ stress test: with 7 classes, the averaging argument predicts the method should struggle. Office-31, where Subspace Alignment and KBTL were evaluated, gives continuity with linear-subspace prior work. NCT-CRC with QuiltNet tests the text-as-source extension against LP+text.
There are three decisive checks. The first is the planted-shortcut experiment: our subspace should place near-zero weight on the source-only feature, while source-only and pooled discriminant analysis should place substantial weight on it. The second is a per-direction audit on real data. Directions retained by source discriminant analysis but rejected by our criterion should, when added back, lower target accuracy. If they do not, the texture-trap explanation is wrong for these benchmarks. The third is a gap-versus-shots curve against the weight-penalty fine-tuning baseline. The claim predicts our advantage is largest at 1 shot and on the largest style gaps, and shrinks by 16 shots. A flat or reversed curve would refute it.
References
- Unsupervised Visual Domain Adaptation Using Subspace Alignment
- Kernelized Bayesian Transfer Learning
- Subspace Identification for Multi-Source Domain Adaptation
- Progressive Mix-Up for Few-Shot Supervised Multi-Source Domain Transfer
- Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning
- Merging SVMs with Linear Discriminant Analysis: A Combined Model
- Learning the Optimal Discriminant SVM With Feature Extraction
- Few-Shot Adaptation of Medical Vision-Language Models
- Multi-source Few-shot Domain Adaptation