TL;DR
This paper introduces ComReg, a novel framework that unifies multi-expert learning and contrastive learning to tackle the challenge of long-tailed image classification. By explicitly coordinating decision boundary optimization and feature learning, ComReg achieves state-of-the-art performance on challenging benchmarks, offering a more robust solution to class imbalance.
What Problem Does This Paper Address?
Image classification models often struggle with "long-tailed" datasets, where a few classes have abundant data (head classes) and many classes have very few samples (tail classes). This imbalance causes models to become biased towards head classes, leading to poor performance on underrepresented tail classes. Existing methods often try to fix this by either redistributing data in the feature space or adjusting decision boundaries, but these approaches can destabilize training or degrade performance on head classes. The core challenge is to simultaneously improve performance on both head and tail classes in a stable and effective manner.
Methods
ComReg proposes a unified framework that integrates several key components:
- Multi-Expert Learning with Online Knowledge Distillation: Inspired by DeiT-LT, ComReg uses a dual-classifier student model. One classifier (tail-class expert) is supervised by ground-truth labels, while the other (head-class expert) is guided by soft targets from a teacher model via knowledge distillation. This setup encourages specialization, with the tail-class expert focusing on minority classes and the head-class expert on majority classes.
- Decoupled Balanced Contrastive Learning (DBCL): To improve feature learning, ComReg employs a contrastive learning approach. It introduces class prototypes derived from the tail-class expert to ensure all classes are represented in mini-batches. Crucially, it decouples positive pairs (anchor, augmented view, class prototype) and assigns distinct weights, allowing for differentiated emphasis and mitigating semantic bias.
- Delayed-Weighted Prototype Learning Strategy: To address the unreliability of prototypes in early training stages, ComReg uses a delayed weighting strategy. This ensures that prototypes are incorporated into the contrastive learning process only after the model has developed sufficient capacity to learn class-level semantics, leading to more stable and robust representation learning.
These components are integrated into a single optimization loop, allowing them to mutually constrain and synergize, leading to a unified regularization effect.
Datasets
The proposed ComReg framework was evaluated on the following datasets:
- CIFAR-LT: This benchmark includes CIFAR10-LT and CIFAR100-LT, constructed with varying imbalance factors (100, 50, and 10) to simulate long-tailed distributions.
- MedMNIST v2: A collection of real-world medical imaging datasets. The authors selected seven representative subsets to ensure diversity in imaging modalities and class imbalance patterns. These subsets cover various medical imaging types like dermoscopy, chest X-rays, and abdominal CT scans.
Key Insights and Ideas
- Unified Optimization is Key: The paper strongly advocates for a unified approach where decision boundary learning and feature space regularization are not treated as independent tasks but are explicitly coordinated. This synergy is the core of ComReg's effectiveness.
- Prototypes as Bridges: Using class prototypes from the tail-class expert within the contrastive learning framework is a clever way to inject decision boundary information directly into feature learning, helping to align these two crucial aspects.
- Delayed Gratification for Prototypes: Recognizing that prototypes are initially unreliable, the delayed weighting strategy is a pragmatic solution that prevents the model from being misled early in training, leading to more stable learning.
- Expert Specialization through Contrastive Learning: The contrastive learning branch, guided by prototypes, helps the two expert heads (head-class and tail-class) to learn more distinct semantic representations, enhancing their specialization.
Key Results
- State-of-the-Art Performance: ComReg consistently outperforms previous state-of-the-art methods on CIFAR10-LT and CIFAR100-LT across various imbalance factors, achieving notable accuracy improvements. For instance, on CIFAR100-LT with p=100, ComReg achieved 54.1% accuracy, outperforming the baseline by 10.1%.
- Strong Performance on Medical Data: On the MedMNIST v2 subsets, ComReg demonstrated strong applicability, achieving the largest performance improvement on the highly imbalanced DM subset and obtaining the highest ACC scores on BL, BR, and PM subsets.
- Improved Feature Representation: t-SNE visualizations show that ComReg-trained models exhibit more compact intra-class clustering and clearer inter-class boundaries compared to baseline and DeiT-LT models.
- Expert Specialization: Grad-CAM visualizations confirm that the tail-class expert focuses on more fine-grained regions, highlighting its specialization in minority-class patterns.
- Stable Training: The delayed weighting strategy contributes to stable performance, with low variance across different random seeds.
Limitations and Future Work
- Computational Cost: While ComReg achieves superior performance, the multi-expert design increases training complexity. The paper notes that the additional computational cost is primarily confined to the training phase.
- Statistical Significance: The authors acknowledge that the current evaluation does not include statistical significance analysis.
- Hyperparameter Sensitivity: While a sensitivity analysis for hyperparameter 'a' was conducted, a more extensive analysis of other hyperparameters could be beneficial.
- Future Directions: The authors suggest that improving computational efficiency, analyzing cost-performance trade-offs, and conducting more comprehensive statistical evaluations are important directions for future work.
Summary updated