TL;DR
Replay buffers in class-incremental learning typically store either raw real samples (which fail under tight memory limits) or distilled synthetic data (which hit diminishing returns as capacity increases). This paper introduces Hybrid Memory Replay, which couples an efficient sliding-window continual data distillation technique with a greedy conditional real-sample selection mechanism. By combining information-dense synthetic exemplars with complementary real samples, the approach consistently outperforms traditional replay buffers across multiple continual learning baselines.
What Problem Does This Paper Address?
In Class Incremental Learning (CIL), neural networks learn new classes in consecutive phases without revisiting previous tasks, frequently triggering catastrophic forgetting. Replay-based approaches mitigate this by caching exemplars from old classes in a bounded memory buffer.
However, practitioners face a dilemma:
- Storing only real exemplars: When buffer limits are tight (e.g., 10–20 samples per class), real samples fail to cover the full data distribution.
- Storing only synthetic exemplars via Data Distillation (DD): While distilled data packs rich representative signals into very few samples, its accuracy gains plateau as buffer size expands due to inherent information loss during optimization.
- Computational bottlenecks: Conventional dataset distillation requires running multiple random-initialization training trajectories—an impractical cost in continual settings where task models follow a single, sequential path.
The authors set out to unify real and synthetic memory into an alternating, complementary framework without inflating computational overhead.
Methods
The proposed framework consists of two core components integrated into an alternating update loop:
[Training on Task-t] ---> [Continual Data Distillation (CDD)] ---> [Conditional Real Data Selection] ---> [Updated Hybrid Memory]
- Continual Data Distillation (CDD): Instead of sampling checkpoints across multiple trajectories, CDD distills synthetic exemplars ($S_t$) from the historical checkpoints ($\theta_{1:N}$) of the current task's single training trajectory. To keep memory bounded, it employs a sliding window of size $\tau$ (e.g., $\tau = 4$), updating synthetic samples only on the most recent checkpoints via distribution matching (DM) or gradient/trajectory matching objectives.
-
Conditional Real Data Selection: After synthesizing $S_t^*$, selecting real exemplars $R_t[A_t]$ is framed as a combinatorial problem: finding the real subset that minimizes the remaining discrepancy between the hybrid memory and the original dataset: $$\arg \min_{A_t \subseteq V_t, |A_t| \le k} \ell([R_t[A_t]; S_t^*], R_t; \theta_t)$$ Because this subset optimization is NP-hard, the authors apply a greedy algorithm to iteratively pick real instances that best complement the existing distilled exemplars.
-
Model Update: When task $t+1$ arrives, the model trains on current real data $R_{t+1}$ alongside the combined hybrid memory $H_{1:t} = H_{1:t-1} \cup \{S_t^*, R_t[A_t^*]\}$.
The authors also provide theoretical bounds showing that under mild contraction assumptions, the performance loss of training on this hybrid buffer compared to full joint training remains strictly bounded across sequential tasks.
Datasets
The authors evaluated their method using ResNet-18 backbones on:
- CIFAR-100 (100 classes, 50,000 train / 10,000 test images).
- TinyImageNet (200 classes, 100,000 train / 10,000 test images).
Both datasets were tested under two standard evaluation protocols with a fixed budget of 20 exemplars per class:
- Zero-base: All classes split evenly across 5 or 10 phases (e.g.,
b0-5,b0-10). - Half-base: Half the classes learned in phase 1, with the remaining split across 5 or 10 phases (e.g.,
half-5,half-10).
Key Insights and Ideas
- Complementarity beats exclusivity: Distilled data efficiently establishes class centroid/distribution statistics, while greedily chosen real instances preserve hard boundary samples and high-frequency patterns that distillation discards.
- Trajectory reuse avoids DD retraining overhead: Generating synthetic exemplars along the actual CIL optimization path is not only drastically faster than multi-trajectory DD, but also aligns the distilled artifacts with the network’s actual evolution.
- A balanced 1:1 ratio is empirically optimal: Ablating synthetic-to-real exemplar splits revealed that performance peaks at a 50/50 balance (e.g., 10 synthetic + 10 real).
- Conditioning is crucial: Greedily selecting real samples conditioned on the distilled set significantly outperforms naive random sample selection + distillation.
Key Results
The hybrid buffer was plugged into three prominent replay algorithms—iCaRL, BEEF, and FOSTER:
- CIFAR-100 Zero-Base 10-Phase:
- FOSTER with hybrid memory (
FOSTER w H) achieved 76.01% Average Incremental Accuracy (AIA) and 66.63% Last Task Accuracy (LAA), improving over vanilla FOSTER (75.00% AIA / 65.12% LAA). iCaRL w Hreached 60.00% AIA, compared to vanilla iCaRL's 57.03% AIA.- CIFAR-100 Half-Base 5-Phase:
- BEEF saw major gains, jumping from 68.57% to 76.31% AIA (+7.74%) and from 61.56% to 70.52% LAA (+8.96%).
- TinyImageNet Zero-Base 10-Phase:
FOSTER w Hreached 68.55% AIA / 56.93% LAA, outperforming the baseline FOSTER (65.37% AIA / 52.70% LAA).- Ablation vs. pure memories: On all base architectures, the hybrid buffer strictly outperformed buffers consisting purely of real exemplars or purely of distilled synthetic exemplars under the same 20-exemplar-per-class constraint.
Limitations and Future Work
- Hand-tuned Hybrid Ratio: The split between synthetic and real exemplars is currently fixed through empirical grid search (set to 0.5). Developing an adaptive mechanism that automatically shifts the ratio based on task complexity or class distribution remains an open challenge.
- Distillation Overhead: Even with a sliding-window checkpoint scheme, running gradient-based dataset distillation (e.g., 10,000 iterations per task) still incurs higher training time than traditional exemplar-storage heuristics like herding.
Summary updated