Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization

Sihao Liu, Yibo Yang, Xiaojie Li, David A. Clifton, Bernard Ghanem

Abstract

Online Continual Learning (OCL) faces significant challenges in balancing plasticity for new tasks and stability for old ones. We present S6MOD, a plug-and-play module that integrates selective State Space Models (SSMs) with a novel "Mixture of Discretization" (MoD) strategy. By employing class-conditional routing based on feature uncertainty, S6MOD dynamically adjusts model capacity, allocating more resources to novel or ambiguous classes while preserving knowledge of well-learned ones. Experimental results on CIFAR and Tiny-ImageNet benchmarks demonstrate that S6MOD consistently achieves state-of-the-art performance by improving feature discriminability and mitigating catastrophic forgetting.

Introduction

In real-world scenarios, data arrives in a continuous stream rather than static batches. Online Continual Learning (OCL) aims to train models on these streams where each data point is seen only once. This paradigm faces two primary hurdles: catastrophic forgetting, where new information overwrites old knowledge, and a lack of plasticity, where static architectures struggle to adapt quickly to new class features. While existing methods often rely on replay buffers to store old examples, they frequently treat the model architecture as a fixed entity. This paper proposes S6MOD, an adaptable architecture that tunes its learning intensity dynamically.

Methodology

S6MOD is designed as a modular branch that can be integrated into existing OCL backbones. Its architecture is founded on three core components:

  • Selective State Space Model (S6): Inspired by the "Mamba" architecture, S6 layers model complex dynamics efficiently. Unlike static weights, S6 parameters adapt based on the input context.
  • Mixture of Discretization (MoD): SSMs rely on discretization to transition between states. S6MOD utilizes a "mixture of experts" approach, allowing the model to select from multiple discretization patterns to capture diverse data dynamics.
  • Class-Conditional Routing: This mechanism maintains feature prototypes for each class to estimate uncertainty. For high-uncertainty (new or confusing) classes, the router activates more discretization patterns ($N_k$) to increase capacity. Conversely, for low-uncertainty (well-known) classes, it uses fewer patterns to maintain stability.
  • Contrastive Discretization Loss ($\mathcal{L}_{Cont}$): This loss function ensures that discretization patterns are consistent within a class but diverse across different classes, enhancing task-specific feature extraction.
The S6MOD framework (a) integrates a parallel branch with a mixture of discretization experts (c) and a class-conditional router (b) to manage model plasticity and stability.

Experiments and Results

S6MOD was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet. Models were restricted to a single pass over the data to simulate a true online environment.

Key Insights

  • Dynamic Capacity Allocation: The model preserves stability for easy classes while increasing plasticity for difficult ones by using uncertainty-driven routing. This balances the learning of new tasks with the preservation of old ones.
  • SSM Effectiveness: The dynamic nature of State Space Models proves highly effective at handling the distribution shifts inherent in continual learning, despite their typical association with long-sequence modeling.

Performance Metrics

S6MOD consistently outperformed baselines across all datasets. On Tiny-ImageNet, adding S6MOD to the "OnPro" baseline resulted in an approximate 4.5% improvement in average accuracy. When combined with modern OCL methods like OCM, S6MOD achieved new state-of-the-art results.

t-SNE visualizations demonstrate that S6MOD (right columns) produces significantly clearer class separation and tighter clusters compared to baseline methods (left columns).

Visual analysis via t-SNE (above) confirms that S6MOD produces much tighter and more distinct class clusters, explaining the significant boost in classification performance.

Conclusion and Future Work

While S6MOD offers a powerful solution for OCL, some limitations remain. Prototype maintenance for the routing mechanism may become computationally expensive in scenarios with millions of classes. Additionally, performance can be sensitive to the weighting of the new loss functions. Future research may extend the "mixture of discretization" concept to other streaming data modalities, such as audio or robotic sensor data.

Summary updated

BibTeX
@inproceedings{liu2025enhancing,
  author = {Sihao Liu and Yibo Yang and Xiaojie Li and David A. Clifton and Bernard Ghanem},
  title = {{Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01909},
}
Figure