Abstract Real-world datasets frequently exhibit long-tailed distributions where a few majority classes dominate, causing standard neural networks to become heavily biased. This research, presented at WACV 2021, identifies the offset (bias) term in the final classification layer as a key "leak" for class-frequency bias. The authors propose a novel de-biasing framework that replaces learned offsets with estimated values derived from feature means and applies a uniform constraint. This approach forces the model to learn more robust representations for minority classes, leading to state-of-the-art performance on long-tailed benchmarks like ImageNet-LT and Places2-LT.
Introduction
In many practical machine learning scenarios, data is naturally imbalanced. A few "majority" classes may have thousands of examples, while "minority" or "tail" classes have only a handful. Standard training objectives, such as minimizing overall risk via Cross-Entropy loss, often lead neural networks to favor these frequent classes. In extreme cases, a model might achieve high overall accuracy while maintaining near-zero accuracy on rare classes, simply by predicting the majority class every time.
While previous research has focused on re-sampling data or re-weighting loss functions, these methods often lead to overfitting on minority samples. This paper identifies a specific architectural culprit: the offset (bias) term in the final linear layer.
The Offset as a Bias Leak
The researchers discovered that the decision boundary between classes in a standard classifier is naturally pushed toward minority classes. This happens because the offset term in the final layer absorbs the class prior (frequency). Effectively, the model penalizes minority classes before it even processes the input features.
As shown in the analysis above, the trained offset closely follows the log-prior of the class distribution. By regulating this bias, the researchers aim to directly tackle the source of the imbalance at the decision-making stage.
Methodology: De-biasing via Estimated Offsets
The authors utilize Gaussian Discriminative Analysis (GDA) to mathematically model how class imbalance affects the classifier. They propose three key components to rectify this bias:
- Empirical Offset: Instead of allowing the network to learn a biased offset, they define an offset based on the mean of the features for each class. Geometrically, this moves the decision boundary to the exact midpoint between class distributions.
- Estimated Offset: Since calculating global feature means during every training step is computationally prohibitive, they introduce a learnable parameter for each class that tracks the feature mean (similar to "center loss").
- Uniform Constraint ($L_{uniform}$): A loss term is added to encourage these offsets to have a uniform magnitude across all classes. This prevents the network from "cheating" by shifting offsets to favor frequent classes, forcing the backbone to instead learn more discriminative features to distinguish between classes.
By de-biasing the classifier, the feature extractor is forced to work harder to find meaningful patterns in minority classes rather than relying on frequency-based shortcuts.
Experimental Results
The proposed method was evaluated on major long-tailed benchmarks, including ImageNet-LT and Places2-LT. The results demonstrate significant improvements in both overall balance and tail-class performance.
Key Performance Metrics
- ImageNet-LT: The method achieved 42.9% overall accuracy with a ResNet-10 backbone, outperforming the baseline (21.9%) and established methods like OLTR (35.6%).
- Few-Shot Gains: The most dramatic improvements occurred in classes with fewer than 20 images. On ImageNet-LT, accuracy for these classes jumped from a baseline of 0.1% to 31.4%.
- Scalability: Using a deeper ResNeXt-152 backbone, the method reached 57.1% accuracy, surpassing the previous state-of-the-art (LWS) by nearly 4%.
Conclusion
By identifying and fixing the "bias leak" in the final layer's offset, this research provides a robust framework for class-imbalanced learning. The method ensures that decision boundaries are placed fairly, which in turn improves the quality of the learned representations. While the approach assumes Gaussian-like feature distributions, its empirical success across diverse datasets suggests it is a powerful tool for real-world applications where data is rarely distributed equally.
Summary updated