Many classifiers built on frozen deep features model each class as a single bell-shaped cloud, a Gaussian with one mean and one covariance, and score new points by how likely they are under each class's cloud. Two separate literatures have found fault with this. The first says the features are not bell-shaped at all: they are skewed and heavy-tailed, so they should be reshaped with a power transform before any Gaussian is fit. The second says a class is often several clouds rather than one, so it should get several prototypes. Current methods stack the two fixes without asking how they interact. They apply one global reshaping to every class and then give every class the same number of prototypes.
This note starts from an observation. A mixture of Gaussians is also a flexible density estimator, so a skewed but single-peaked class will be fit better by three components than by one. Part of the gain credited to "multimodality" may therefore be the mixture approximating a skewed shape that the global transform failed to correct for that particular class. We propose to separate these two effects. Each class gets its own invertible per-dimension reshaping, with the likelihood corrected for the change of variables. Extra mixture components are added only when the reshaped class still shows several peaks. The result is a classifier whose number of components per class is decided by the data, together with a measurement of how much intra-class multimodality in standard benchmarks survives proper per-class normalization.
Core Claim
Global Gaussianization followed by fixed-count clustering mixes up two things: non-Gaussian shape and genuine sub-types. A class-specific Yeo-Johnson transform (a monotone power-like map, one parameter per feature dimension, that makes a variable more bell-shaped) should be fit per class, and its Jacobian included in the class likelihood. Mixture components should then be added only where a model-selection test on the transformed features demands them. We predict three outcomes. First, most classes on general-object benchmarks will turn out to be unimodal after per-class transformation. Second, the classes that remain multimodal will match semantically recognizable sub-types. Third, the resulting classifier will match or beat multi-prototype methods while using fewer components and yielding a more compact in-distribution region.
Concretely, for class $c$ with per-dimension transform $T_c$ (Yeo-Johnson parameters $\lambda_{c,d}$ for dimension $d$), mixture weights $\pi_{c,k}$, means $\mu_{c,k}$ and covariances $\Sigma_{c,k}$ over $K_c$ components, the class score is
$$ \log p(x \mid c) = \log \sum_{k=1}^{K_c} \pi_{c,k}\,\mathcal{N}\big(T_c(x);\mu_{c,k},\Sigma_{c,k}\big) + \sum_{d} \log T_{c,d}'(x_d). $$
The second term is the log-derivative of the transform. For non-negative post-ReLU features it equals $(\lambda_{c,d}-1)\log(1+x_d)$. Without it, likelihoods computed in differently warped spaces cannot be compared across classes. $K_c$ is chosen per class by the Bayesian information criterion on $T_c(x)$, with covariance shrinkage as in FeCAM.
Relation to Prior Work
The method draws on three sources. Free Lunch for Few-shot Learning: Distribution Calibration and FeCAM established that a Tukey square-root transform makes frozen features Gaussian enough for covariance-based classification. FeCAM adds class-specific covariances with shrinkage, which reached 70.8% exemplar-free 10-task accuracy on CIFAR-100. Both use one global exponent and one Gaussian per class. On the Importance of Gaussianizing Representations supplies the Yeo-Johnson transform with a fast Newton-step estimate of its parameter. That paper uses it inside training as a normalization layer. We use it post hoc and per class, which that paper does not do.
The closest competitor is FeNeC. It applies a global Tukey transform, runs k-means with a fixed number of clusters for every class, and classifies by Mahalanobis distance. It reports 79.59% average incremental accuracy on ImageNet-Subset, more than one point above FeCAM. FeNeC motivates us, because it shows that multiple centroids help. It also leaves our question open, because it cannot tell whether its centroids capture sub-types or just patch skew, and it notes that extra centroids hurt on simpler data.
On the multimodality side, Prototypical Clustering Networks for dermatology and the Gaussian mixture VAE for open-set recognition show real sub-type structure and the open-space cost of forcing one convex region. Both fix component counts in advance and do no Gaussianization. Mahalanobis++ constrains our design. It shows that feature-norm variation corrupts Gaussian scoring and that ℓ2 projection fixes much of it, so any per-class transform must be tested with and without ℓ2 projection rather than assumed to replace it.
Research Program
The work has three parts.
The first is a diagnostic study. For each class on CIFAR-100, ImageNet-Subset, CUB-200 and Dermnet features from the frozen-backbone protocols of FeCAM and FeNeC, we compute the BIC-selected component count under four conditions: raw features, global Tukey, global Yeo-Johnson, and per-class Yeo-Johnson. The quantity of interest is how the distribution of selected $K_c$ shifts across these four conditions. As a positive control, we check on Dermnet whether surviving components line up with known presentation variants. As a synthetic calibration, we use skewed unimodal and genuinely bimodal classes, and check that the pipeline recovers $K=1$ and $K=2$ respectively.
The second is the classifier above, evaluated in exemplar-free class-incremental learning. Our transform and mixture statistics are per class, so new classes never disturb old ones. This is the property FeCAM and FeNeC rely on.
The third is a secondary open-set check. Here we score out-of-distribution inputs by the maximum class log-likelihood and test whether adaptive components shrink the in-distribution region compared with a single inflated covariance.
A practical risk is estimation. Per-dimension Yeo-Johnson parameters are univariate fits and cheap. Mixture covariances are not. We therefore share one shrunk covariance across a class's components when the class has few samples, and we report sensitivity to that choice.
Baselines
The naive reference is a nearest-class-mean Euclidean classifier on raw frozen features, plus FeCAM as the strong single-Gaussian method. For FeCAM we reuse its protocol and aim to reproduce its reported 70.8% on CIFAR-100 10-task before comparing.
The strongest existing competitor is FeNeC. It already combines normalization, multiple centroids and Mahalanobis scoring, and it holds the best number in the file on ImageNet-Subset: 79.59% average incremental accuracy. The file reports no isolated CIFAR-100 number for FeNeC, so on CIFAR-100 we will rerun it ourselves. Every method gets the same backbone, the same class splits and the same held-out tuning budget over its own hyperparameters: exponent, cluster count, shrinkage. Beating FeNeC means higher average incremental accuracy across three seeds on both benchmarks, or equal accuracy with a clearly smaller mean $K_c$.
The key-ingredient ablations are two. The first uses a per-class transform but drops the Jacobian term; we expect this to hurt, since class scores then live in incomparable spaces. The second applies the per-class transform with $K_c$ forced to one, which isolates the transform from the mixture.
For the open-set check, the baselines are Mahalanobis++, which reports an average FPR@95 reduction of 7.6% over standard Mahalanobis across 44 ImageNet models, and ORA (OOD Detection with Relative Angles), which reports 38.34% average FPR95 across nine ImageNet backbones. We compare on the same backbones.
Why not just…?
Why not just run FeNeC with the cluster count chosen per class by BIC? This is the real threat, and it is exactly our "global transform plus adaptive $K$" condition. If it matches our full method on accuracy and on selected component counts, then per-class Gaussianization adds nothing and the claim fails. The diagnostic study settles this directly. The claim survives only if per-class transforms systematically lower the selected $K_c$ and improve held-out class log-likelihood at equal or better accuracy.
Why not just use a larger Gaussian mixture, since enough components approximate any shape? A mixture can mimic skew, but only by spending components and covariance parameters that are poorly estimated from a few hundred samples. It also scatters small components into the tail, which enlarges the in-distribution region. We will compare at a matched parameter count. If a plain mixture wins there, our argument about parsimony is wrong.
Why not just ℓ2-normalize as Mahalanobis++ does? ℓ2 projection removes norm variation but not per-dimension skew within a class, and Mahalanobis++ keeps one mean per class. On the sphere, the mean of a multimodal class lies off the data. We test ℓ2 alone, per-class Yeo-Johnson alone, and both combined. If ℓ2 alone closes the gap in both tasks, that is an honest negative result for the transform.
Why not just a per-class normalizing flow? In exemplar-free incremental learning each class has few samples in hundreds of dimensions, and a flexible flow would overfit. Per-dimension Yeo-Johnson is the smallest invertible flow that fixes marginal skew. That is the point of choosing it, and the forced-$K_c{=}1$ ablation measures how far it gets alone.
Proposed Experiments
The decisive check is the diagnostic histogram of selected $K_c$ under the four transform conditions, paired with held-out per-class log-likelihood. If per-class transformation collapses most CIFAR-100 and ImageNet-Subset classes to one component but leaves Dermnet and CUB classes multimodal, the two sources of apparent multimodality are separated. That pattern is expected because Dermnet has documented presentation sub-types, with PCN at 49.6% versus 37.1% mean per-class accuracy for single prototypes.
The accuracy checks come second. They are exemplar-free class-incremental learning on CIFAR-100 and ImageNet-Subset against FeCAM, FeNeC and BIC-FeNeC; 10-shot Dermnet mean per-class accuracy against PCN's 49.6%; and near-OOD FPR95 against Mahalanobis++ and ORA. Two further outcomes would each falsify part of the claim. If BIC-FeNeC ties the full method on every one of these tasks, the per-class transform adds nothing. If the Jacobian-free ablation performs as well as the full method, the likelihood correction is unnecessary.
References
- Free Lunch for Few-shot Learning: Distribution Calibration
- FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning
- On the Importance of Gaussianizing Representations
- FeNeC: Enhancing Continual Learning via Feature Clustering with Neighbor- or Logit-Based Classification
- Few-Shot Learning for Dermatological Disease Diagnosis
- Open-Set Recognition with Gaussian Mixture Variational Autoencoders
- Mahalanobis++: Improving OOD Detection via Feature Normalization
- OOD Detection with Relative Angles