Abstract
Imbalanced data is ubiquitous in machine learning, particularly in medical and fine-grained image datasets. Existing continual learning methods often rely on classification accuracy, which is poorly suited for imbalanced settings. We propose DIANA (Model DecouplIng ANd Alignment), a framework designed to effectively maximize the Area Under the ROC Curve (AUC) in lifelong learning. By decoupling the learning of current tasks from the retention of old knowledge into two separate models that gradually align, DIANA addresses the gradient interference problem inherent in imbalanced data streams. Our results demonstrate that DIANA achieves state-of-the-art performance across multiple imbalanced benchmarks.
Introduction
Lifelong learning (or continual learning) aims to enable AI models to learn a sequence of tasks without "catastrophic forgetting." While most research assumes balanced datasets and uses classification accuracy as the primary metric, real-world applications like medical diagnosis or fraud detection often involve heavy class imbalance (e.g., 99:1). In such scenarios, accuracy is misleading; a model predicting only the majority class achieves high accuracy but zero utility.
Area Under the ROC Curve (AUC) is the preferred metric for imbalanced data. However, directly optimizing for AUC in a lifelong learning setup is challenging. The authors identify a gradient interference problem: in imbalanced settings, the gradient direction required to learn new tasks often conflicts sharply with the direction needed to preserve old knowledge. This conflict leads to poor convergence and forgetting.
Methodology: The DIANA Framework
To mitigate gradient interference, we introduce DIANA (Model DecouplIng ANd Alignment). Unlike traditional single-model approaches, DIANA maintains two sets of weights simultaneously.
- Model Decoupling: DIANA utilizes two models, $\mathbf{w}$ and $\mathbf{v}$. Model A ($\mathbf{w}$) focuses on maximizing AUC for the current task, while Model B ($\mathbf{v}$) focuses on maximizing AUC for previous tasks using a replay buffer.
- Model Alignment: To ensure eventual convergence to a single model, an alignment penalty ($\beta \cdot \text{dist}(\mathbf{w}, \mathbf{v})$) is added to the loss function. This allows the models to "negotiate" shared knowledge without direct gradient collisions.
- Minimax Reformulation: AUC optimization typically requires pairwise comparisons, which is computationally expensive. DIANA employs a minimax saddle point reformulation, enabling efficient updates using individual samples while maintaining global AUC optimization.
Experimental Setup
DIANA was evaluated on five benchmarks, including three standard sets modified for imbalance and two real-world imbalanced datasets:
- Modified Standard Sets: Split-CIFAR100, Split-CUB200, and Split-AWA2, where the positive class was reduced to 5%.
- Real-World Imbalanced Sets: ISIC2019 (skin lesion images) and EuroSat (satellite land cover).
Results and Discussion
DIANA achieved state-of-the-art performance across all imbalanced benchmarks. Key insights include:
- Geometric Interference: Analysis showed that gradients for new and old tasks are often "obtuse" (pointing away) in imbalanced data, explaining why standard memory-based methods fail.
- Performance Gains: DIANA improved average AUC by 9.0% on Split-CUB, 13.0% on Split-AWA2, and 6.2% on EuroSat compared to competitive baselines.
- Efficiency: Despite the dual-model architecture, DIANA remains computationally efficient, outperforming regularization methods like EWC and GEM in training speed.
Conclusion
DIANA provides a robust solution for imbalanced lifelong learning by directly optimizing AUC and managing gradient interference through model decoupling. While it requires more memory than single-model approaches, its superior performance and efficiency make it a compelling choice for real-world applications. Future work will explore extending DIANA to class-incremental learning scenarios where task boundaries are not explicitly defined.
Summary updated