Trust-Region Class Weighting: Up-Weight Rare Classes Only Where Their Geometry Is Stably Estimated

When a classifier is trained on data where a few classes are common and many are rare, the standard fix is class weighting: multiply the loss of rare-class examples by a large factor so they are not drowned out. The weight is a single number per class, and it amplifies every direction of that class's gradient equally. With only a handful of training images per class, though, most of those directions are estimated from sampling noise. A rare class's feature cloud has a few directions that reliably separate it from its neighbors and many that simply reflect which five or twenty images happened to be sampled. Scalar weighting pushes on both kinds equally. We think this is a concrete mechanism behind the "distribution gap" failure of reweighting. Geometry of Long-Tailed Representation Learning shows that reweighted training separates empirical tail samples while the true population clouds still overlap.

We propose replacing the scalar class weight with a directional one. The amplification is applied only inside the part of each class's feature space that is statistically stable, and the class is left at ordinary weight elsewhere. Stability is decided by the spectral gap: the distance between the eigenvalues of a class's feature covariance that we keep and the ones we discard. Classical perturbation theory says a subspace estimated from finite samples can be trusted only when this gap clearly exceeds the size of the sampling noise. The contribution is to use that century-old criterion as the rule deciding where class weights may act, and to test whether it beats simply turning the weights down.

Core Claim

For each class $c$, let $\hat\Sigma_c$ be the covariance of its penultimate-layer features, computed from a running feature bank, with eigenvalues $\hat\lambda_1 \ge \hat\lambda_2 \ge \dots$. Davis–Kahan-type bounds say how far the estimated top-$k$ eigenvector subspace $\hat U_{c,k}$ can rotate away from the true one. The rotation angle $\theta$ satisfies $\sin\theta \le \|E_c\| / (\hat\lambda_k - \hat\lambda_{k+1})$, where $E_c$ is the gap between the sample and population covariance. Because $\|E_c\|$ shrinks roughly like $\sqrt{r/n_c}$ for effective rank $r$ and $n_c$ samples (standard covariance concentration), a tail class supports only a small stable dimension $k_c$. A head class supports a large one.

We modify the gradient that a class-$c$ example sends back into the feature extractor, $g_i$, as follows:

$$ \tilde g_i = g_i + (w_c - 1)\,\hat U_{c,k_c}\hat U_{c,k_c}^\top g_i , $$

Here $w_c$ is any standard class weight, such as inverse frequency or effective number, and $k_c$ is the largest index whose eigengap passes a bootstrap stability test. Outside the stable subspace, the example contributes with weight one. We apply the same projection to the feature vector that enters the classifier-weight gradient.

The claim is that this recovers most of the tail-class benefit of heavy weighting while avoiding its two known costs. The first cost is overfitting tail training points, which we measure as the train-to-test drift of tail class centers. The second is the high gradient variance that destabilizes early training.

The theory we reuse comes from three papers. Zwald et al. show that eigenspace estimation error in kernel PCA is controlled by the gap $\lambda_k - \lambda_{k+1}$ rather than by ambient dimension. Mavroeidis shows that a subspace survives sampling noise exactly when the gap dominates the perturbation norm, and turns this into a selection rule. Our stability test is a direct descendant of that rule. Neither paper concerns supervised training or class imbalance. Our contribution is to use their criterion inside the loss.

Relation to Prior Work

Two lines of work connect representation geometry to imbalance, and both motivate this proposal without doing it.

The first line studies neural collapse, the tendency of late-stage features to shrink onto their class means. Imbalance Trouble derives the terminal geometry under imbalance from the singular value decomposition of the label matrix. It shows that minority classifiers shrink and that convergence slows as imbalance grows. Rethinking Loss Reweighting as an Inverse Problem infers per-batch scalar weights that equalize per-class losses and so drive class means toward a symmetric simplex. Both treat each class as a point, its mean. We treat each class as a subspace with an uncertain estimate. Wang et al.'s weights remain scalar and act on all directions, so their method is a candidate $w_c$ to plug into our projection, not a competitor to the projection itself.

The second line measures per-class spectra directly. Balanced Data, Imbalanced Spectra shows on frozen encoders that the scale of a class's covariance spectrum predicts its error. The correlation of spectral offset with accuracy is −0.68, and their Spectral Quantile Score correlates at 0.90 with class bias. They fix this at test time through augmentation choice. We act during training, and we use the gap rather than the spectral scale. FeatRecon regularizes tail covariance toward head covariance and synthesizes tail features, whereas we generate nothing. Group Spectral Regularization penalizes the top singular value of class-conditioning parameters in GANs, bounding a norm where we measure a gap.

One paper constrains the design. Zhong et al.'s calibration study shows that naive end-to-end reweighting damages features and calibration. Our method must therefore beat reweighting on calibration as well as accuracy, or it has not addressed the problem it names.

Research Program

The work has three parts.

Diagnostic. Before any method, we test the premise. We train a standard ResNet-32 on CIFAR-100-LT at imbalance factors 10, 50, 100 and 200, with and without inverse-frequency weighting. We measure how much reweighting moves tail features, and we decompose that movement into the component inside versus outside each class's bootstrap-stable subspace. We also measure the drift between train and test class centers in each component. The premise predicts that reweighting's damage to held-out tail features lives mostly in the unstable complement. If the damage is instead spread evenly, the directional idea has no reason to work and we stop.

Method. We maintain a per-class feature bank of the last few epochs, which costs little at feature dimension 64 to 512. Once per epoch we eigendecompose each class covariance. We choose $k_c$ by resampling the bank and keeping the largest $k$ whose subspace angle across resamples stays below a fixed threshold. We then apply the projected weighting above. The threshold is the only new hyperparameter, and it is set once on a held-out split.

Two issues need care. First, classes with very few samples may get $k_c = 0$. In that case they revert to unweighted training in all directions, which is itself a testable prediction: for the rarest classes, weighting the class mean direction alone may be what matters, so we also test always including the mean-difference direction. Second, the research file notes that global gaps vanish under non-uniform densities. Computing the gap per class rather than globally is our answer to that.

Analysis. We track, over training, the eigengaps of tail classes and of pairs of adjacent tail classes. We ask whether directional weighting delays the imbalance threshold at which tail centers merge, as described in FeatRecon's analysis. Spectral clustering theory (Ng et al.; Sun et al.) ties that merging to a vanishing gap of the affinity-graph Laplacian. We measure that gap on a class-affinity graph built from features as a secondary readout.

Baselines

Plain cross-entropy. This is the naive reference point. Zhong et al. report 41.2% top-1 and 27.5% ECE for it on CIFAR-100-LT at imbalance factor 100 (Improving Calibration for Long-Tailed Recognition). We rerun it in our codebase so every comparison shares architecture, augmentation, epochs and seeds.

Scalar class-balanced weighting. We run both inverse-frequency and effective-number weights, which are the direct parent of our method and use the same $w_c$ we plug in. No reported result was found on Lacuna for effective-number weighting on CIFAR-100-LT, so our own run is the reference.

Loss-equalizing inverse reweighting of Wang et al. This is the strongest weighting method in the file. Rethinking Loss Reweighting as an Inverse Problem reports a 6.26 to 7.08 point top-1 gain over cross-entropy across imbalance ratios, but the file gives no absolute figure. We run it with its authors' schedule. We then run it again with its weights passed through our projection, which is the cleanest test of whether directionality adds anything on top of the best scalar weights.

MiSLAS. This two-stage method reports 47.0% top-1 and 4.83% ECE on CIFAR-100-LT at imbalance factor 100 (Zhong et al.) and is the calibration bar.

FeatRecon. This is the strongest accuracy number in the file: 52.5% at imbalance factor 100 and 72.9% on iNaturalist 2018 (FeatRecon). It relies on feature synthesis and contrastive loss, so we compare both against it directly and against our projection combined with its contrastive component.

Ablations isolating the key ingredient. Three ablations isolate what matters. The first projects onto a random subspace of the same dimension $k_c$. The second uses the top-$k_c$ eigenvectors with $k_c$ fixed by eigenvalue rank instead of the stability test. The third uses the stable subspace of a different class.

What counts as success. Beating scalar weighting and Wang et al. at matched tuning budget means a statistically reliable gain in few-shot-class accuracy over five seeds, without loss in many-shot accuracy, and with lower ECE. We do not claim to beat FeatRecon outright. The target is to close part of the gap to it without synthetic features, and to add to it when combined.

Why not just…?

Why not just temper the weights, for example $w_c \propto n_c^{-1/2}$ or a cap? This is the most serious threat. Tempering also reduces gradient variance and tail overfitting, and it costs nothing. The difference is that tempering shrinks amplification uniformly, including along the few stable directions where full amplification helps. We therefore sweep the tempering exponent and cap with the same tuning budget as our threshold and report the best setting. If the best tempered scalar matches the projection on few-shot accuracy and ECE, the directional mechanism is unnecessary and we will say so.

Why not just use Wang et al.'s loss-equalizing weights? Their weights choose how much to amplify each class, not where in feature space to apply it. The file notes they need a delayed activation because their geometric target only emerges late in training. Our wrapped variant tests directly whether restricting their weights to stable subspaces lets them act earlier and more safely.

Why not just decouple training and rebalance the classifier afterward? Zhong et al. find that decoupled methods stay badly miscalibrated. FeatRecon argues that tail features beyond a critical imbalance have already merged during stage one, where no classifier fix can separate them. The imbalance-factor sweep, run up to 200, shows whether our method delays that merging. MiSLAS is included so the comparison is made rather than assumed.

Why not just borrow head-class covariance for the tail, as FeatRecon does? Borrowing assumes the tail class shares the head's shape. The stability test instead uses only what the tail's own data supports. The two ideas are compatible, and the combined run tells us whether they overlap.

Proposed Experiments and Decisive Checks

The main benchmarks are CIFAR-10-LT and CIFAR-100-LT across imbalance factors 10 to 200, ImageNet-LT, and iNaturalist 2018. We report top-1 accuracy split into many-, medium- and few-shot classes, ECE, and the train-test drift of tail class centers. We also report gradient-norm variance over the first epochs, to test the claimed stabilization.

A balanced-data control uses full CIFAR-100 with per-class spectral weights set by gap stability instead of sample count. It asks whether the mechanism also helps the spectrally hard classes of Kaushik et al., where frequency-based weights are exactly one.

Three results would settle the proposal. First, the diagnostic must show that reweighting's held-out tail damage concentrates in unstable directions. Second, the stability-tested projection must beat both the random-subspace and fixed-rank ablations, showing that the gap criterion itself matters. Third, the projection must beat the best-tuned tempered scalar weights on few-shot accuracy and ECE. If the first check fails, the premise is wrong. If the second fails, the method reduces to a generic low-rank trick. If the third fails, the simpler alternative wins. In each case the negative result would still tell us where reweighting's harm actually lives.