De-biasing Neural Networks with Estimated Offset for Class Imbalanced Learning

Byungju Kim, Hyeong Gwon Hong, Junmo Kim

Abstract Real-world datasets frequently exhibit long-tailed distributions where a few majority classes dominate, causing standard neural networks to become heavily biased. This research, presented at WACV 2021, identifies the offset (bias) term in the final classification layer as a key "leak" for class-frequency bias. The authors propose a novel de-biasing framework that replaces learned offsets with estimated values derived from feature means and applies a uniform constraint. This approach forces the model to learn more robust representations for minority classes, leading to state-of-the-art performance on long-tailed benchmarks like ImageNet-LT and Places2-LT.

Introduction

In many practical machine learning scenarios, data is naturally imbalanced. A few "majority" classes may have thousands of examples, while "minority" or "tail" classes have only a handful. Standard training objectives, such as minimizing overall risk via Cross-Entropy loss, often lead neural networks to favor these frequent classes. In extreme cases, a model might achieve high overall accuracy while maintaining near-zero accuracy on rare classes, simply by predicting the majority class every time.

While previous research has focused on re-sampling data or re-weighting loss functions, these methods often lead to overfitting on minority samples. This paper identifies a specific architectural culprit: the offset (bias) term in the final linear layer.

The Offset as a Bias Leak

The researchers discovered that the decision boundary between classes in a standard classifier is naturally pushed toward minority classes. This happens because the offset term in the final layer absorbs the class prior (frequency). Effectively, the model penalizes minority classes before it even processes the input features.

Analysis of class-frequency bias: (a) shows the long-tailed prior distribution; (b) demonstrates how standard trained offsets naturally mirror this prior; (c) shows the target empirical offset used for de-biasing.

As shown in the analysis above, the trained offset closely follows the log-prior of the class distribution. By regulating this bias, the researchers aim to directly tackle the source of the imbalance at the decision-making stage.

Methodology: De-biasing via Estimated Offsets

The authors utilize Gaussian Discriminative Analysis (GDA) to mathematically model how class imbalance affects the classifier. They propose three key components to rectify this bias:

  1. Empirical Offset: Instead of allowing the network to learn a biased offset, they define an offset based on the mean of the features for each class. Geometrically, this moves the decision boundary to the exact midpoint between class distributions.
  2. Estimated Offset: Since calculating global feature means during every training step is computationally prohibitive, they introduce a learnable parameter for each class that tracks the feature mean (similar to "center loss").
  3. Uniform Constraint ($L_{uniform}$): A loss term is added to encourage these offsets to have a uniform magnitude across all classes. This prevents the network from "cheating" by shifting offsets to favor frequent classes, forcing the backbone to instead learn more discriminative features to distinguish between classes.

By de-biasing the classifier, the feature extractor is forced to work harder to find meaningful patterns in minority classes rather than relying on frequency-based shortcuts.

Experimental Results

The proposed method was evaluated on major long-tailed benchmarks, including ImageNet-LT and Places2-LT. The results demonstrate significant improvements in both overall balance and tail-class performance.

Accuracy across class indices on Places2-LT: The proposed method (red) maintains significantly higher accuracy across minority classes compared to the baseline (blue).

Key Performance Metrics

  • ImageNet-LT: The method achieved 42.9% overall accuracy with a ResNet-10 backbone, outperforming the baseline (21.9%) and established methods like OLTR (35.6%).
  • Few-Shot Gains: The most dramatic improvements occurred in classes with fewer than 20 images. On ImageNet-LT, accuracy for these classes jumped from a baseline of 0.1% to 31.4%.
  • Scalability: Using a deeper ResNeXt-152 backbone, the method reached 57.1% accuracy, surpassing the previous state-of-the-art (LWS) by nearly 4%.

Conclusion

By identifying and fixing the "bias leak" in the final layer's offset, this research provides a robust framework for class-imbalanced learning. The method ensures that decision boundaries are placed fairly, which in turn improves the quality of the learned representations. While the approach assumes Gaussian-like feature distributions, its empirical success across diverse datasets suggests it is a powerful tool for real-world applications where data is rarely distributed equally.

Summary updated

Abstract

The imbalanced distribution of the training data makes the networks biased to the frequent classes. Existing methods to resolve the problem involve re-sampling, reweighting, or cost-sensitive learning. Most of them anticipate that emphasizing the minority classes during the training would help the network to learn better representations. In this paper, we propose a method for reparameterizing softmax classifiers’ offsets so that training is less sensitive to class imbalance. We first observe that the trained offset of the baseline linear classifier is biased toward the majority classes due to the imbalance. Instead of the trained offset, we define the estimated offset, and constrain it to be uniform over the classes. In experiments with long-tailed benchmarks, our method exhibits the best performance. These experiments verify that our proposed method effectively encourages the networks to learn better representations for minority classes while preserving the performance for the majority classes.
BibTeX
@inproceedings{kim2021de,
  author = {Byungju Kim and Hyeong Gwon Hong and Junmo Kim},
  title = {{De-biasing Neural Networks with Estimated Offset for Class Imbalanced Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00152},
}
Figure