Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization

Sihao Liu, Yibo Yang, Xiaojie Li, David A. Clifton, Bernard Ghanem

Introduction

In real-world scenarios, data is rarely static. Instead, it often arrives in a continuous, non-stationary stream where samples are seen only once—a paradigm known as Online Continual Learning (OCL). Training models in this environment presents two fundamental challenges: Catastrophic Forgetting, where new information overwrites previously learned knowledge, and Poor Adaptability, where static architectures struggle to capture the complex, evolving dynamics of the data stream.

While many OCL strategies rely on experience replay buffers to mitigate forgetting, they often overlook the architectural plasticity required to adapt to new features. This paper introduces S6MOD, a novel plug-and-play module that integrates Selective State Space Models (S6) with a "Mixture of Discretization" strategy to bridge this gap.

Methodology: The S6MOD Framework

S6MOD is designed as an auxiliary branch that can be integrated into existing OCL backbones, such as ResNet. The core innovation lies in its ability to dynamically adjust learning capacity based on the complexity of the incoming data.

Selective State Space Model (S6) Branch

Borrowing from the "Mamba" architecture, the module utilizes an S6 branch to model long-range dependencies. Unlike standard convolutions or transformers, S6 uses input-aware weights, making the model highly responsive to the specific characteristics of the current data stream.

Mixture of Discretization (MoD) and Routing

In State Space Models, the discretization parameter ($Δ$) acts as a control knob, determining the balance between current input and historical state. S6MOD replaces a single fixed $Δ$ with a Sparse Mixture of Experts (MoE) approach.

The S6MOD architecture features (b) Class-Conditional Routing to determine model capacity, (c) a Mixture of Discretization within the S6 block, and (d) a Contrastive Discretization Loss to refine feature separation.

As shown in Figure 2, the system employs Class-Conditional Routing to manage stability and plasticity:

  • High Uncertainty: When a class is easily confused with others (calculated via distance between feature prototypes), the router allocates more discretization patterns ($N_k$), providing higher capacity to learn complex boundaries.
  • Low Uncertainty: For well-defined classes, fewer patterns are used, preserving stability and preventing unnecessary parameter updates that could lead to forgetting.

To further enhance discriminative power, a Contrastive Discretization Loss ($L_{Cont}$) is applied. This loss ensures that discretization patterns for the same class are pulled together while patterns for different classes are pushed apart, creating a more robust feature space.

Experimental Evaluation

The researchers evaluated S6MOD across three standard OCL benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet. These tests were conducted across various memory buffer sizes (from $M=500$ to $M=10,000$) to assess efficiency under different storage constraints.

Key Results

S6MOD was tested as an enhancement to seven state-of-the-art OCL methods, including ER, OCM, and OnPro. The results were consistently positive:

  • Performance Gains: The module provided a significant accuracy boost across all methods. For instance, it improved the OnPro method by approximately 4.2% on CIFAR-100 and 4.5% on Tiny-ImageNet.
  • State-of-the-Art Accuracy: When paired with top-tier methods, S6MOD achieved record-breaking results, such as 79.51% on CIFAR-10.
  • Visualizing Feature Separation: The impact of S6MOD is clearly visible in feature space visualizations.
t-SNE visualizations demonstrate that adding S6MOD to base methods like OnPro and OCM results in significantly tighter and more distinct class clusters, reducing inter-class interference.

As illustrated in Figure 6, the t-SNE plots show that S6MOD helps the model produce much cleaner class separations compared to the base methods alone. This improved clustering directly correlates with the reduced forgetting rates observed in the quantitative results.

Discussion and Future Work

S6MOD demonstrates that treating the discretization parameter as a learnable mixture is a powerful way to handle multi-scale temporal features in streaming data. By intelligently routing resources to "difficult" classes, the model mimics a more human-like focus on challenging information.

However, some limitations remain. The addition of an auxiliary MoE branch increases computational overhead (FLOPs) and parameter count. Furthermore, the reliance on class prototypes could present a memory bottleneck in systems with thousands of classes. Future work may focus on optimizing the routing efficiency or extending the "Mixture of Discretization" concept to other domains like video-based learning or complex time-series analysis.

Summary updated

Abstract

Online continual learning (OCL) seeks to learn new tasks from data streams that appear only once, while retaining knowledge of previously learned tasks. Most existing methods rely on replay, focusing on enhancing memory retention through regularization or distillation. However, they often overlook the adaptability of the model, limiting the ability to learn generalizable and discriminative features incrementally from online training data.To address this, we introduce a plug-and-play module, S6MOD, which can be integrated into most existing methods and directly improve adaptability. Specifically, S6MOD introduces an extra branch after the backbone, where a mixture of discretization selectively adjusts parameters in a selective state space model, enriching selective scan patterns such that the model can adaptively select the most sensitive discretization method for current dynamics.We further design a class-conditional routing algorithm for dynamic, uncertainty-based adjustment and implement a contrastive discretization loss to optimize it. Extensive experiments combining our module with various models demonstrate that S6MOD significantly enhances model adaptability, leading to substantial performance gains and achieving the state-of-the-art results. The code is available at https://github.com/MyToumaKazusa/S6MOD.
BibTeX
@inproceedings{liu2025enhancing,
  author = {Sihao Liu and Yibo Yang and Xiaojie Li and David A. Clifton and Bernard Ghanem},
  title = {{Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01909},
}
Figure