Overfitting of Neural Nets Under Class Imbalance: Analysis and Improvements for Segmentation

Zeju Li, Konstantinos Kamnitsas, Ben Glocker

Abstract

Deep learning models often overfit in data-scarce, class-imbalanced scenarios typical of medical imaging. This paper demonstrates that such overfitting is asymmetric: while the majority class remains stable, the minority class "drifts" toward the decision boundary on unseen data, severely impacting sensitivity. We introduce asymmetric variants of popular training techniques—Large Margin Loss, Focal Loss, Adversarial Training, and Mixup—that prioritize the protection of the under-represented class. Our results show significant boosts in segmentation performance, particularly when training data is extremely limited.

Introduction

Convolutional Neural Networks (CNNs) are the standard for medical image segmentation but require substantial annotated data to generalize. In clinical practice, data is often scarce, and structures of interest (like tumors or small organs) frequently occupy less than 1% of the total volume. Standard regularization techniques often fail in these imbalanced settings. This research identifies that overfitting in these contexts manifests as a specific failure mode: precision remains high, but sensitivity plummets as the model becomes increasingly "shy" about predicting the minority class on new patients.

The Logit Shift Phenomenon

The core insight of this work is the "logit shift." By analyzing the raw scores (logits) before the final softmax layer, the authors found a stark difference between training and test distributions. For the majority background class, the logit distributions for training and test data are nearly identical. However, for the minority foreground class, the test activations shift significantly toward the decision boundary.

Visualization of the logit shift: as training data decreases, the foreground test distribution (red) drifts toward the decision boundary, while the background (blue) remains stable.

This suggests that while the network learns "generic" filters for the ubiquitous background, it "memorizes" the rare foreground samples using specific filters that fail to generalize. Consequently, the decision boundary, which appears safe during training, becomes too restrictive for unseen foreground structures.

Asymmetric Regularization Methods

To counteract the logit shift, the authors propose modifying standard machine learning "tricks" to be asymmetric, specifically "shielding" the minority class from the decision boundary.

Conceptual comparison between original (symmetric) and proposed asymmetric regularizers.
  1. Asymmetric Large Margin Loss: Adds a penalty margin only to the minority class, forcing the network to learn more robust, distant features for it.
  2. Asymmetric Focal Loss: Down-weights "easy" samples only for the majority class, ensuring the network maintains focus on the minority class even if it appears easy during training.
  3. Asymmetric Adversarial Training: Generates difficult perturbed examples exclusively for the minority class to expand its representation in the feature space.
  4. Asymmetric Mixup: Blends images but assigns a "hard" minority label to any mix containing a significant portion of the rare class, preventing the boundary from encroaching on the foreground.

Experimental Evaluation

The methods were tested on two tasks: Brain Tumor Core Segmentation (BraTS) and Small Organ Segmentation (whole-body MRI). Data scarcity was simulated by using as little as 5% of the available training sets.

The asymmetric modifications consistently outperformed symmetric baselines. In the most extreme case (5% brain tumor data), Asymmetric Mixup achieved a Dice Similarity Coefficient (DSC) of 0.59, compared to 0.51 for standard Cross-Entropy. A combination of all four asymmetric techniques further boosted the DSC to 0.62, with sensitivity jumping from 0.43 to 0.66. Notably, while symmetric regularizers sometimes degraded performance compared to simple baselines, the asymmetric versions provided a consistent "safety net" and performance boost.

Conclusion

Symmetry in regularization can be detrimental in class-imbalanced segmentation. By acknowledging the asymmetric nature of logit drift, we can design more effective training protocols. Future work will explore extending these principles to multi-class segmentation and refining hyperparameter selection for these asymmetric losses.

Summary updated

BibTeX
@inproceedings{li2019overfitting,
  author = {Zeju Li and Konstantinos Kamnitsas and Ben Glocker},
  title = {{Overfitting of Neural Nets Under Class Imbalance: Analysis and Improvements for Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  doi = {10.1007/978-3-030-32248-9\_45},
}
Figure