In the ideal world of computer vision research, datasets are massive, perfectly balanced, and meticulously labeled. In the real world, data is often "deteriorated"—riddled with class imbalances, tiny sample sizes, and noisy labels. At CVPR 2021, Takumi Kobayashi introduced a deceptively simple yet powerful way to help Deep Convolutional Neural Networks (CNNs) thrive in these harsh conditions by rethinking how we measure similarity in the final classification layer.
The Problem: Broad Support and Intra-Class Variance
Standard CNNs typically use a softmax layer where the "score" for a class is based on the dot product (or cosine similarity) between a feature vector and a learned class weight. While effective on clean data like ImageNet, this approach has a "broad support region." It doesn't strongly penalize features for drifting far from their class center, as long as they are on the "correct side" of the decision boundary.
On deteriorated datasets, this lack of constraint leads to high intra-class variance and overfitting. Existing solutions, like Center Loss, attempt to mitigate this by adding extra terms to the loss function to pull features toward class centers. However, these often require complex hyperparameter tuning and extra computational overhead. The proposed method seeks to bake this regularization directly into the similarity measure itself.
Proposed Method: t-vMF Similarity
The core innovation is moving beyond cosine similarity to a measure based on the von Mises-Fisher (vMF) distribution, a fundamental distribution in directional statistics that can be thought of as a Gaussian distribution wrapped around a sphere.
From vMF to t-vMF
Initially, a similarity based on the standard vMF distribution creates a "compact support" function that focuses heavily on the area directly around the class weight. However, a pure vMF distribution has "light tails," meaning the gradient vanishes almost instantly for samples that aren't already very close to the center. This makes training unstable.
Borrowing an idea from t-SNE, the author replaces the exponential profile with a student-t distribution profile. This "t-vMF" similarity is compact enough to force features together but has "heavy tails" that allow the network to still learn from outliers without the gradients disappearing.
The final formula can be implemented as a simple one-line change to the logit calculation:
# Standard Cosine: phi = cosine
# t-vMF Similarity:
phi = (1 + cosine) / (1 + kappa * (1 - cosine)) - 1
Here, $κ$ (kappa) is a single hyperparameter that controls how "compact" the class distribution should be. While a positive $κ$ compresses classes (ideal for noisy/small data), a negative $κ$ expands the support region, which can increase inter-class discriminativity on healthy datasets.
Experimental Evaluation
The method was rigorously tested across several deteriorated scenarios, including imbalanced sets (ImageNet-LT, iNaturalist), small-scale subsets (ImageNet-S/SS), and noisy labels (ImageNet-N).
Key Results
The t-vMF similarity consistently outperformed standard softmax and popular regularization techniques like Dropout, Center Loss, and Label Smoothing:
- Noisy Labels: On ImageNet-N, t-vMF ($κ=64$) achieved a 69.19% error rate, a massive improvement over standard Softmax (82.34%) and significantly better than Virtual Softmax (72.40%).
- Imbalanced Data: On ImageNet-LT, it reduced the error rate from 61.32% (Softmax) to 56.31%.
- Efficiency: Because it is a simple algebraic modification of the logits, it adds virtually zero computational cost during training or inference.
Conclusion and Future Work
t-vMF similarity provides an elegant, implicit regularization by changing the geometry of the similarity measure. While $κ$ is an intuitive hyperparameter, its optimal value currently depends on the degree of data deterioration. Future research may explore automatic tuning of $κ$ or its synergy with data-level augmentations like Mixup.
Summary updated