The strongest few-shot image classifiers do not learn how to learn. They train an ordinary classifier on many labeled base classes, freeze it, and classify new classes from one to five examples using a simple rule, such as distance to each class's average feature. That simple rule works only if the frozen features already separate the new classes. Standard cross-entropy training on base classes works against this. Late in training, it squeezes the features toward the few directions that distinguish the base classes and discards everything else, a failure known as neural collapse. The discarded directions may be exactly the ones a new class needs.
We propose to leave the base-class directions to cross-entropy and to protect everything else separately. At each training step, we take the part of each image's feature that lies outside the span of the base-class classifier weights. On that part only, we apply VICReg, a self-supervised loss that keeps each dimension varied across images, keeps dimensions uncorrelated, and keeps two augmented views of the same image close. The leftover space then stays high-dimensional and tied to real image content rather than augmentation noise. The claim is that novel classes should be separable in it.
Core Claim
Two results in the file frame the idea. Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning shows that few-shot accuracy with a frozen extractor is predicted by one quantity. That quantity is the within-class variance measured only along the axis between two class means, normalized by the distance between them, called directional CDNV (class-distance normalized variance). Total within-class variance does not predict accuracy. The paper derives bounds for nearest-centroid and linear probes from this quantity, but it proposes no training objective. Variance-Covariance Regularization Improves Representation Learning (VCReg) adds variance and covariance penalties to supervised pre-training at intermediate layers. It argues that the invariance term is redundant there because cross-entropy already enforces invariance, and it improves ResNet-50 average transfer across nine datasets from 62.33% to 67.96%. It never evaluates few-shot benchmarks and never looks at decision-axis geometry.
Our position is that, for few-shot transfer, both halves of VCReg's design need to change. First, the regularizer should act only on the orthogonal complement of the base-class span, because a variance floor imposed inside that span fights the within-class compression that cross-entropy correctly performs there. Second, the invariance term is not redundant in the complement. Cross-entropy says nothing about directions it cannot see, so a variance floor alone would fill the complement with augmentation noise. That noise is precisely the variance that raises directional CDNV for novel classes. The supporting evidence is the mechanism in Understanding Dimensional Collapse in Contrastive Self-supervised Learning: when augmentation variance along a direction exceeds data variance, that direction is useless and gets pruned. The falsifiable prediction follows directly. Complement-only VICReg should lower novel-class directional CDNV and raise 1-shot accuracy more than either global VICReg or complement variance/covariance without invariance.
Research Program
Let $h_i \in \mathbb{R}^D$ be the penultimate feature of image $i$, and let $W \in \mathbb{R}^{C \times D}$ hold the classifier weights for the $C$ base classes. The complement projector is $P_\perp = I - W^\top (W W^\top)^{-1} W$, computed with gradients stopped through $W$ and refreshed at every step. A small expander $g$ maps $P_\perp h_i$ to $z_i$. Two augmented views give $z_i$ and $z'_i$. The training loss is
$$\mathcal{L} = \mathrm{CE}(W h_i, y_i) + \alpha\left[\lambda\, s(Z,Z') + \mu\,(v(Z)+v(Z')) + \nu\,(c(Z)+c(Z'))\right],$$
where $s$, $v$ and $c$ are the invariance, variance-hinge and off-diagonal covariance terms of VICReg. Following that paper, we start from $\lambda=25, \mu=25, \nu=1$ and tune only the overall weight $\alpha$. At test time we discard $g$, freeze the network, and classify with nearest centroid or logistic regression on the full feature $h$, following the protocol of Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?.
Alongside accuracy, we will build a diagnostic suite from the directional CDNV definition. It measures three things on novel classes. The first is directional CDNV computed on the full feature. The second is the fraction of each novel class-mean difference that lies inside the base span versus the complement. The third is the effective rank of the complement features. These measurements will test whether plain cross-entropy actually starves the complement, which is the premise of the whole proposal. They will also test whether any accuracy gain comes through the predicted mechanism.
Baselines
The naive reference point is Classifier-Baseline: cross-entropy pre-training followed by a frozen nearest-centroid probe. Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning reports 58.91% 5-way 1-shot accuracy on miniImageNet for it and 63.17% for Meta-Baseline. The strongest simple representation baseline is the distilled embedding of Rethinking Few-Shot Image Classification, which reports 82.14% 5-shot accuracy on miniImageNet.
The strongest existing pre-training method in the file is MGRCL from Rethinking the Sample Relations for Few-Shot Classification. It adds a consistency loss across augmentations of the same image and a memory-bank contrastive loss between classes. It reports 69.57% (1-shot) and 84.41% (5-shot) on miniImageNet, and 86.14% (1-shot) on CUB-200-2011. It is the most direct competitor because it also fixes what supervised pre-training compresses, but it does so through sample relations rather than subspace geometry. STL DeepBDC from Joint Distribution Matters, at 67.83% miniImageNet 1-shot, is a second strong reference.
VCReg applied to few-shot is an essential baseline, but no reported result was found on Lacuna for VCReg on any few-shot benchmark, so we will run it ourselves.
Fairness requires that every method be re-run under one protocol. All methods share a ResNet-12 backbone, the same augmentations, the same epoch count and the same frozen-probe protocol. Each gets an equal hyperparameter search over the same number of configurations, selected on the validation classes. We report MGRCL's published numbers alongside our re-runs, because the file does not state its backbone.
Beating the field means a statistically separated gain, with non-overlapping 95% confidence intervals over 2,000 test episodes, over re-run MGRCL and VCReg in 1-shot on miniImageNet and tieredImageNet. We aim to at least match the 69.57% figure. We will also run ablations: global VICReg on all of $h$, complement-only variance/covariance without invariance, and the complement projector replaced by a random subspace of equal dimension.
Why not just…?
Why not just add a VICReg auxiliary loss to the whole feature? This is the most serious alternative, and it may work nearly as well. Our argument against it is that a variance floor inside the base span resists cross-entropy's useful compression, so it should cost base-class separability. The global-VICReg ablation settles the question directly. If it matches complement-only VICReg on novel accuracy and on directional CDNV, the subspace idea is unnecessary and we will report that plainly.
Why not just use VCReg as published? VCReg drops invariance and regularizes all intermediate dimensions. Our prediction is that in the complement, variance without invariance raises novel directional CDNV, because the preserved variance is partly augmentation noise. The complement-variance/covariance-only ablation, together with the CDNV measurements, tests this.
Why not just stop training early or use self-distillation? Both reduce late-stage collapse. Rethinking Few-Shot Image Classification shows that distillation helps, and Meta-Baseline documents novel-class accuracy peaking during training. Neither method adds structure to the unused dimensions; each only slows their loss. We will compare against the best early-stopping checkpoint and against a distillation generation. We will also test whether our method is additive with distillation.
Proposed Experiments
We pre-train on base classes of miniImageNet, tieredImageNet, CUB-200-2011 and FC100, and evaluate 5-way 1-shot and 5-shot with both probes. For cross-domain transfer we evaluate miniImageNet→CUB and miniImageNet→Cars, where STL DeepBDC reports a gain of over 4 points. We use Meta-Dataset as a stretch evaluation.
The first decisive check tests the premise. For a cross-entropy-only model, we measure how much novel class-mean difference lies in the complement. If it is negligible, the complement cannot matter and the project stops. The second check tests the mechanism. Across all methods and checkpoints, novel-class directional CDNV should track 1-shot accuracy, and complement-only VICReg should achieve the lowest CDNV while costing nothing in base-class accuracy. The third check tests specificity. Complement-only VICReg must beat both global VICReg and the random-subspace control. Otherwise the benefit is generic regularization rather than the protection of unused directions.
We also expect the gain to be larger in 1-shot than in 5-shot, and larger on tieredImageNet's semantically distant splits than on miniImageNet. Both follow from the view that novel directions matter more when base and novel classes share less structure. If this pattern fails to appear, it would weaken the proposed explanation even if average accuracy improves.
References
- Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning
- Variance-Covariance Regularization Improves Representation Learning
- Understanding Dimensional Collapse in Contrastive Self-supervised Learning
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?
- Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
- Rethinking the sample relations for few-shot classification
- Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification