Steering Fixed Classifier Frames Away from Lighting Directions in Long-Tailed Vision

A common fix for long-tailed image classification freezes the last layer of the network. Long-tailed means a few classes have most of the images. The frozen layer is a set of maximally spread-out class directions, called a simplex equiangular tight frame (ETF): one unit vector per class, with every pair equally far apart. Freezing it stops frequent classes from crowding rare ones out of the feature space. This works, but it raises a question nobody has answered. Every rotation of the frame fits the training data equally well, yet different rotations generalize differently. Random fixed frames give "identical training convergence but drastically different test generalization" (Feature Directions Matter). Rotated frames shift CIFAR-10 validation accuracy by up to 1.73% with nothing else changed, and the authors of that result call the cause "open for future investigation" (A Study of Neural Collapse Phenomenon).

We propose a concrete, testable answer and a method that follows from it. A frame orientation generalizes badly when its class directions overlap the directions in feature space along which features move under nuisance changes in lighting, meaning color temperature and white balance. If that holds, the cure is to measure those lighting directions and place the frame orthogonal to them, rather than training the backbone to erase lighting. The lighting directions can be estimated from abundant head classes and then protect rare tail classes, which have too few images to learn lighting invariance on their own. Rare classes are often photographed under idiosyncratic conditions, so the effect should be largest exactly where long-tailed methods are weakest.

Core Claim

Let $f(x) \in \mathbb{R}^d$ be the penultimate feature of image $x$. Let $g_T$ re-light the image to color temperature $T$ along the physically realistic black-body (Planckian) curve, as in Planckian jitter. Define the lighting covariance $\Sigma_L = \mathbb{E}_{x,T}\big[(f(g_T x) - f(x))(f(g_T x) - f(x))^\top\big]$. Let $U_k$ be its top-$k$ eigenvectors, the lighting subspace. For a class-direction matrix $M \in \mathbb{R}^{d \times C}$ (an ETF times a rotation), define the lighting overlap

$$ \Omega(M) = \frac{\lVert U_k^\top M \rVert_F^2}{\lVert M \rVert_F^2}, $$

the fraction of the frame's energy lying in directions that lighting moves.

The claim has two parts. First, across equally good-on-training rotations, $\Omega$ predicts test accuracy, and predicts it more strongly under lighting-shifted test sets. Second, building the frame inside the orthogonal complement of $U_k$ gives lighting robustness to tail classes without making the backbone colorblind. The backbone may keep encoding color and illuminant, which fine-grained categories need; the classifier simply cannot read the lighting component.

There is room for this whenever $d \ge (C-1) + k$. For a ResNet on CIFAR-100, $d = 512$ against $C - 1 = 99$. For ResNet-50 on ImageNet-LT, $d = 2048$ against $C - 1 = 999$.

Relation to Prior Work

The rotation question is motivated by two papers. A Study of Neural Collapse Phenomenon documents that frame orientation matters but offers no mechanism. Feature Directions Matter supplies a method we reuse, called RBL. RBL learns a rotation $R \in \mathrm{SO}(d)$ of a fixed ETF by gradient steps through the matrix exponential, which keeps $R$ exactly orthogonal. RBL picks the rotation through the training loss, which by construction cannot tell isometric frames apart well. We pick it through an explicit, measurable nuisance criterion, and we test whether RBL's learned rotations already drift toward low $\Omega$.

The frozen-frame foundation comes from Inducing Neural Collapse in Imbalanced Learning, which fixes the ETF and trains with a squared-error "dot-regression" loss. All-around Neural Collapse adds the requirement that class feature means, not only classifier weights, align with the frame. That paper's pull of class means toward fixed targets is exactly what could bake a tail class's shared lighting cast into its prototype, a risk none of these papers examine.

On the lighting side, two papers supply the nuisance model. Planckian jitter re-lights images along black-body temperatures. What Else Can Fool Deep Learning? emulates realistic camera white-balance errors. Both pursue invariance through augmentation and never look at feature geometry. The Planckian work also shows that aggressive color invariance makes representations colorblind, which constrains our design: we avoid suppressing lighting in the backbone. Flatness is Necessary, Neural Collapse is Not and Optimizer Choice Matters caution that collapse geometry is neither necessary for generalization nor guaranteed under AdamW. We therefore treat optimizer choice as an axis to report rather than assume, and we do not claim symmetry itself causes generalization. Our claim is narrower: frame orientation relative to a nuisance subspace matters.

Research Program

The work has three stages.

The first stage is diagnostic. On balanced CIFAR-10 and CIFAR-100 we train a ResNet with fixed ETF and dot-regression under 20 to 30 random rotations, with seeds otherwise matched. At convergence we estimate $\Sigma_L$ from the trained backbone using Planckian re-lighting of training images. We then correlate $\Omega$ with clean test accuracy and with accuracy on white-balance-perturbed test images generated with the Afifi–Brown procedure.

A circularity worry is that the backbone adapts to the frame. To address it, we also compute $\Omega$ using a lighting subspace taken from a separate reference backbone that has a learnable classifier. That way the predictor exists before the rotation is chosen. Two controls test whether the effect is specific to lighting: the same overlap computed for random-crop and scale augmentations, and for random subspaces of equal dimension.

The second stage is the method, which we call Lighting-Orthogonal Frame (LOF). After a short warm-up with a learnable classifier, we estimate $U_k$ from head-class images only. We then build the ETF inside the complement of $U_k$ and train with dot-regression. Every few epochs we re-estimate $U_k$ and apply a small RBL-style rotation update whose objective adds $\lambda\,\Omega(M)$, so the frame tracks a drifting lighting subspace without abrupt target jumps. The rank $k$ is picked by the eigenvalue spectrum of $\Sigma_L$ on held-out data. LOF adds no inference cost, unlike the two-backbone latent combination of Planckian jitter.

The third stage builds a benchmark for the joint failure. No benchmark in the file combines label imbalance with lighting bias. We construct lighting-biased versions of CIFAR-100-LT and ImageNet-LT. In training, each tail class is rendered under a single randomly assigned color temperature or white-balance error, while head classes keep natural variation. The test set is rendered under the full lighting range. This isolates the case where a tail class can use its lighting cast as a shortcut.

Baselines

The naive reference is a standard learnable linear classifier with cross-entropy and the usual torchvision color jitter. It shows how much any of this matters. Gap 4 of the research file reports that cross-entropy reaches 42.4% on CIFAR-100-LT at imbalance ratio 0.005 in Neural Collapse To Multiple Centers.

The fixed-ETF baseline is a random fixed frame with dot-regression (Inducing Neural Collapse in Imbalanced Learning). It is the direct parent of LOF and differs only in orientation, so it isolates the claim. RBL reports this fixed-ETF baseline at 51.2% overall and 27.2% few-shot on ImageNet-LT.

The strongest orientation competitor is RBL itself (Feature Directions Matter), reported at 53.3% overall and 34.2% few-shot on ImageNet-LT, and 53.1% on CIFAR-100-LT at ratio 100. The strongest overall long-tailed method in the file is AllNC (All-around Neural Collapse), at 56.5% on ImageNet-LT and 51.50% on CIFAR-100-LT at imbalance factor 200. Because AllNC adds losses on features and class means that are orthogonal to orientation, we also run AllNC with its frame placed by LOF.

Every frame-based method is also run with Planckian jitter plus white-balance augmentation applied to all classes. This is the augmentation-only competitor. For robustness to white-balance shift, the file's reference point is Afifi–Brown: CIFAR-10 accuracy under incorrect white balance rises from 65.5% to 78.6% with their augmentation. We found no reported result on Lacuna for any long-tailed method evaluated under lighting shift, nor for the lighting-biased long-tailed setting. We will therefore reproduce all baselines ourselves.

All runs share the same backbone, epochs, augmentation pipeline (except where augmentation is the variable), and a hyperparameter search of equal size per method. We report SGD and AdamW separately. Beating the field means two things. On lighting-biased benchmarks, LOF combined with augmentation must improve few-shot accuracy under shifted test lighting over RBL combined with the same augmentation, by more than seed-to-seed standard deviation over at least three seeds. On clean ImageNet-LT and CIFAR-100-LT, it must not fall below RBL's reported numbers.

The key ablations place the frame orthogonal to three other subspaces of the same rank: a random subspace, the crop/scale nuisance subspace, and a lighting subspace estimated from all classes rather than head classes only. The random and crop subspaces test whether lighting specificity matters. The head-only versus all-class comparison tests whether tail classes really borrow head statistics.

Why not just…?

Why not just apply Planckian jitter or white-balance augmentation and keep a fixed ETF or AllNC? This is the most serious threat. Augmentation applied to tail images does supply lighting variety, and Afifi–Brown show it works on balanced data. Two reasons it may fall short. First, a tail class with a handful of images, trained toward a fixed target, can still absorb its residual cast. Second, invariance learned in the backbone is the colorblindness that Planckian jitter warns about. We settle this directly with the augmentation-only arm on the lighting-biased benchmark. We also run a linear probe for intrinsic color and illuminant on frozen features. LOF should match augmentation on shifted-test tail accuracy while retaining more color information. If augmentation plus fixed ETF matches LOF on both, the method is unnecessary, and we would report the diagnostic result alone.

Why not just add a feature-consistency loss that pulls each image's feature toward that of its re-lit copy? This erases lighting from the features, which is the opposite design choice. It risks the same color loss, and it does not use the frame to share lighting statistics from head to tail. We include it as a baseline on the same benchmark and compare both robustness and color-probe accuracy.

Why not just trust RBL, since it already learns the rotation? RBL may already find low-overlap frames implicitly. We measure $\Omega$ along RBL's training trajectory. If RBL converges to low overlap on its own, our contribution becomes an explanation of why it works plus a cheaper, data-driven initialization, and we will say so. If it does not, which is plausible because the training loss is nearly flat across rotations, the explicit penalty should add tail robustness.

Why not just white-balance every input with a color-constancy step before classification? Afifi and Brown show that color-constancy failures are themselves the error source, so preprocessing inherits them. We include an off-the-shelf correction-then-classify pipeline as a reference on the white-balance-shifted test sets.

Decisive Checks

The mechanism stands or falls on the first stage. If $\Omega$, computed from the reference backbone, shows no reliable negative correlation with shifted-test accuracy across rotations, the explanation is wrong. If the crop or random-subspace controls predict accuracy just as well, the explanation is wrong. In either case the method should not be built.

If the correlation holds, the method's value rests on two comparisons on lighting-biased CIFAR-100-LT and ImageNet-LT: tail accuracy under shifted lighting versus RBL plus augmentation, and the head-only versus all-class subspace ablation. A secondary check repeats everything under AdamW. Optimizer Choice Matters predicts that AdamW never reaches the collapsed geometry, so we expect the orientation effect to weaken. Confirming or refuting that tells practitioners whether frame orientation is worth tuning in modern training recipes.