Abstract
Class incremental learning (CIL) is a critical requirement for deploying deep learning models in evolving clinical environments where new diagnostic categories emerge over time. However, standard neural networks suffer from catastrophic forgetting, and strict privacy regulations such as HIPAA and GDPR often preclude the use of rehearsal-based methods that store patient data. We present Continual Class-Specific Impression (CCSI), a novel data-free framework designed specifically for medical imaging. CCSI synthesizes "impressions" of previously learned classes using model inversion and a specialized normalization layer, allowing the model to retain old knowledge without storing original images. Our results demonstrate that CCSI significantly outperforms existing data-free methods and achieves performance competitive with rehearsal-based approaches on both MedMNIST and complex echocardiography datasets.
Introduction
In the medical domain, the discovery of new diseases or the refinement of diagnostic categories necessitates models that can adapt incrementally. The primary challenge in this setting is catastrophic forgetting, where training on new data leads to an abrupt loss of performance on previously learned tasks. While rehearsal-based strategies—which replay a subset of old data—are effective in natural image processing, they are often non-viable in healthcare due to the massive storage requirements of high-resolution scans and, more importantly, stringent data privacy laws.
Existing "data-free" incremental learning methods attempt to synthesize data to replace stored samples, but they often struggle with the high dimensionality and subtle patterns characteristic of medical imagery. CCSI addresses these challenges by leveraging the inherent structural consistency of medical scans to generate high-fidelity class impressions.
Methodology
CCSI utilizes a two-stage pipeline consisting of data synthesis and model updating.
1. Data Synthesis via Model Inversion
To recover knowledge of previous classes without access to the original data, CCSI employs Model Inversion. This process treats the frozen, previously trained model as a teacher and optimizes a batch of pixels to "excite" the model’s internal representations of a specific class.
Two key innovations enhance this synthesis:
- Medical Prior Initialization: Unlike standard methods that initialize synthesis from random noise, CCSI starts with a mean image of the class. Because medical images of the same organ share consistent anatomical landmarks, this initialization provides a structural foundation that significantly improves the quality of the generated impression.
- Continual Normalization (CN): Standard Batch Normalization layers are often overwritten by new data statistics, leading to the degradation of synthesized images. CCSI introduces Continual Normalization to preserve the statistics of previous tasks, ensuring that the synthesized "impressions" remain faithful to the original data distribution.
2. Model Update and Loss Functions
The model is updated using a combination of new real data and the synthesized impressions of old data. To bridge the "reality gap" between synthetic and real samples, the authors introduce three specific loss components:
- Intra-domain Contrastive (IdC) Loss: This aligns the feature distributions of synthetic and real data, ensuring the model treats them as part of the same underlying class representation.
- Margin Loss: This establishes a clear mathematical boundary between old and new classes, preventing overlap in the feature space.
- Cosine-Normalized Cross-Entropy: This prevents the model from developing a bias toward new classes, which typically have more (or higher quality) samples during the update phase.
Experimental Results
The framework was rigorously evaluated on two benchmarks: MedMNIST (comprising BloodMNIST, PathMNIST, OrganaMNIST, and TissueMNIST) and a large-scale Heart Echo Dataset containing over 11,000 echocardiography cine series.
Performance Gains
CCSI demonstrated substantial improvements over existing state-of-the-art (SOTA) data-free methods:
- BloodMNIST: CCSI achieved 79.10% accuracy, a massive leap from the 33.20% achieved by the previous leading method, ABD.
- Heart Echo Dataset: In this complex, high-resolution clinical scenario, CCSI reached 83.66% accuracy, while other data-free methods failed to exceed 32%.
Remarkably, CCSI performed comparably to—and in some cases better than—rehearsal methods that actually store and reuse old patient images, proving that high-quality synthetic impressions can effectively replace sensitive raw data.
Discussion and Key Insights
The success of CCSI highlights several critical insights for medical AI:
- Normalization as Memory: Normalization layers act as a repository for a network's data distribution. By protecting these layers through Continual Normalization, we prevent the erasure of "statistical memory."
- Impressions vs. Replicas: The goal of CCSI is not to reconstruct specific patient images, which would pose a privacy risk. Instead, it creates a "class impression" that captures the essential features required to maintain decision boundaries.
- The Power of Priors: Leveraging the structured nature of medical imaging (e.g., anatomical landmarks) allows for more effective synthesis than generic noise-based approaches used in natural image processing.
Limitations and Future Work
Despite its performance, CCSI faces challenges regarding computational cost, as model inversion requires thousands of optimization epochs per task. Additionally, while the mean image initialization provides structure, the synthesized images may lack the full diversity of a real-world patient population. Future research could explore incorporating class variance into the synthesis process and adapting the framework for dynamic architectures that add capacity as new tasks are introduced.
Summary updated