Hybrid Memory Replay: Blending Real and Distilled Data for Class Incremental Learning

Jiangtao Kong, Jiacheng Shi, Shaohan Hu, Ashley Gao, Tianyi Zhou, Huajie ShaoSubmission to ICLR 2025 · Withdrawn

TL;DR

Replay buffers in class-incremental learning typically store either raw real samples (which fail under tight memory limits) or distilled synthetic data (which hit diminishing returns as capacity increases). This paper introduces Hybrid Memory Replay, which couples an efficient sliding-window continual data distillation technique with a greedy conditional real-sample selection mechanism. By combining information-dense synthetic exemplars with complementary real samples, the approach consistently outperforms traditional replay buffers across multiple continual learning baselines.

What Problem Does This Paper Address?

In Class Incremental Learning (CIL), neural networks learn new classes in consecutive phases without revisiting previous tasks, frequently triggering catastrophic forgetting. Replay-based approaches mitigate this by caching exemplars from old classes in a bounded memory buffer.

However, practitioners face a dilemma:

  • Storing only real exemplars: When buffer limits are tight (e.g., 10–20 samples per class), real samples fail to cover the full data distribution.
  • Storing only synthetic exemplars via Data Distillation (DD): While distilled data packs rich representative signals into very few samples, its accuracy gains plateau as buffer size expands due to inherent information loss during optimization.
  • Computational bottlenecks: Conventional dataset distillation requires running multiple random-initialization training trajectories—an impractical cost in continual settings where task models follow a single, sequential path.

The authors set out to unify real and synthetic memory into an alternating, complementary framework without inflating computational overhead.

Methods

The proposed framework consists of two core components integrated into an alternating update loop:

[Training on Task-t] ---> [Continual Data Distillation (CDD)] ---> [Conditional Real Data Selection] ---> [Updated Hybrid Memory]
  1. Continual Data Distillation (CDD): Instead of sampling checkpoints across multiple trajectories, CDD distills synthetic exemplars ($S_t$) from the historical checkpoints ($\theta_{1:N}$) of the current task's single training trajectory. To keep memory bounded, it employs a sliding window of size $\tau$ (e.g., $\tau = 4$), updating synthetic samples only on the most recent checkpoints via distribution matching (DM) or gradient/trajectory matching objectives.
  2. Conditional Real Data Selection: After synthesizing $S_t^*$, selecting real exemplars $R_t[A_t]$ is framed as a combinatorial problem: finding the real subset that minimizes the remaining discrepancy between the hybrid memory and the original dataset: $$\arg \min_{A_t \subseteq V_t, |A_t| \le k} \ell([R_t[A_t]; S_t^*], R_t; \theta_t)$$ Because this subset optimization is NP-hard, the authors apply a greedy algorithm to iteratively pick real instances that best complement the existing distilled exemplars.

  3. Model Update: When task $t+1$ arrives, the model trains on current real data $R_{t+1}$ alongside the combined hybrid memory $H_{1:t} = H_{1:t-1} \cup \{S_t^*, R_t[A_t^*]\}$.

The authors also provide theoretical bounds showing that under mild contraction assumptions, the performance loss of training on this hybrid buffer compared to full joint training remains strictly bounded across sequential tasks.

Datasets

The authors evaluated their method using ResNet-18 backbones on:

  • CIFAR-100 (100 classes, 50,000 train / 10,000 test images).
  • TinyImageNet (200 classes, 100,000 train / 10,000 test images).

Both datasets were tested under two standard evaluation protocols with a fixed budget of 20 exemplars per class:

  1. Zero-base: All classes split evenly across 5 or 10 phases (e.g., b0-5, b0-10).
  2. Half-base: Half the classes learned in phase 1, with the remaining split across 5 or 10 phases (e.g., half-5, half-10).

Key Insights and Ideas

  • Complementarity beats exclusivity: Distilled data efficiently establishes class centroid/distribution statistics, while greedily chosen real instances preserve hard boundary samples and high-frequency patterns that distillation discards.
  • Trajectory reuse avoids DD retraining overhead: Generating synthetic exemplars along the actual CIL optimization path is not only drastically faster than multi-trajectory DD, but also aligns the distilled artifacts with the network’s actual evolution.
  • A balanced 1:1 ratio is empirically optimal: Ablating synthetic-to-real exemplar splits revealed that performance peaks at a 50/50 balance (e.g., 10 synthetic + 10 real).
  • Conditioning is crucial: Greedily selecting real samples conditioned on the distilled set significantly outperforms naive random sample selection + distillation.

Key Results

The hybrid buffer was plugged into three prominent replay algorithms—iCaRL, BEEF, and FOSTER:

  • CIFAR-100 Zero-Base 10-Phase:
  • FOSTER with hybrid memory (FOSTER w H) achieved 76.01% Average Incremental Accuracy (AIA) and 66.63% Last Task Accuracy (LAA), improving over vanilla FOSTER (75.00% AIA / 65.12% LAA).
  • iCaRL w H reached 60.00% AIA, compared to vanilla iCaRL's 57.03% AIA.
  • CIFAR-100 Half-Base 5-Phase:
  • BEEF saw major gains, jumping from 68.57% to 76.31% AIA (+7.74%) and from 61.56% to 70.52% LAA (+8.96%).
  • TinyImageNet Zero-Base 10-Phase:
  • FOSTER w H reached 68.55% AIA / 56.93% LAA, outperforming the baseline FOSTER (65.37% AIA / 52.70% LAA).
  • Ablation vs. pure memories: On all base architectures, the hybrid buffer strictly outperformed buffers consisting purely of real exemplars or purely of distilled synthetic exemplars under the same 20-exemplar-per-class constraint.

Limitations and Future Work

  • Hand-tuned Hybrid Ratio: The split between synthetic and real exemplars is currently fixed through empirical grid search (set to 0.5). Developing an adaptive mechanism that automatically shifts the ratio based on task complexity or class distribution remains an open challenge.
  • Distillation Overhead: Even with a sliding-window checkpoint scheme, running gradient-based dataset distillation (e.g., 10,000 iterations per task) still incurs higher training time than traditional exemplar-storage heuristics like herding.

Summary updated

Abstract

Incremental learning (IL) aims to acquire new knowledge from current tasks while retaining knowledge learned from previous tasks. Replay-based IL methods store a set of exemplars from previous tasks in a buffer and replay them when learning new tasks. However, there is usually a size-limited buffer that cannot store adequate real exemplars to retain the knowledge of previous tasks. In contrast, data distillation (DD) can reduce the exemplar buffer's size, by condensing a large real dataset into a much smaller set of more information-compact synthetic exemplars. Nevertheless, DD's performance gain on IL quickly vanishes as the number of synthetic exemplars grows. To overcome the weaknesses of real-data and synthetic-data buffers, we instead optimize a hybrid memory including both types of data. Specifically, we propose an innovative modification to DD that distills synthetic data from a sliding window of checkpoints in history (rather than checkpoints on multiple training trajectories). Conditioned on the synthetic data, we then optimize the selection of real exemplars to provide complementary improvement to the DD objective. The optimized hybrid memory combines the strengths of synthetic and real exemplars, effectively mitigating catastrophic forgetting in Class IL (CIL) when the buffer size for exemplars is limited. Notably, our method can be seamlessly integrated into most existing replay-based CIL models. Extensive experiments across multiple benchmarks demonstrate that our method significantly outperforms existing replay-based baselines. Our source code is available at https://anonymous.4open.science/r/DD4CIL-510C/.
BibTeX
@misc{kong2024hybrid,
  author = {Jiangtao Kong and Jiacheng Shi and Shaohan Hu and Ashley Gao and Tianyi Zhou and Huajie Shao},
  title = {{Hybrid Memory Replay: Blending Real and Distilled Data for Class Incremental Learning}},
  year = {2024},
  eprint = {2410.15372},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.15372},
}

Loading related content...

Figure