Introduction
In real-world scenarios, data is rarely static. Instead, it often arrives in a continuous, non-stationary stream where samples are seen only once—a paradigm known as Online Continual Learning (OCL). Training models in this environment presents two fundamental challenges: Catastrophic Forgetting, where new information overwrites previously learned knowledge, and Poor Adaptability, where static architectures struggle to capture the complex, evolving dynamics of the data stream.
While many OCL strategies rely on experience replay buffers to mitigate forgetting, they often overlook the architectural plasticity required to adapt to new features. This paper introduces S6MOD, a novel plug-and-play module that integrates Selective State Space Models (S6) with a "Mixture of Discretization" strategy to bridge this gap.
Methodology: The S6MOD Framework
S6MOD is designed as an auxiliary branch that can be integrated into existing OCL backbones, such as ResNet. The core innovation lies in its ability to dynamically adjust learning capacity based on the complexity of the incoming data.
Selective State Space Model (S6) Branch
Borrowing from the "Mamba" architecture, the module utilizes an S6 branch to model long-range dependencies. Unlike standard convolutions or transformers, S6 uses input-aware weights, making the model highly responsive to the specific characteristics of the current data stream.
Mixture of Discretization (MoD) and Routing
In State Space Models, the discretization parameter ($Δ$) acts as a control knob, determining the balance between current input and historical state. S6MOD replaces a single fixed $Δ$ with a Sparse Mixture of Experts (MoE) approach.
As shown in Figure 2, the system employs Class-Conditional Routing to manage stability and plasticity:
- High Uncertainty: When a class is easily confused with others (calculated via distance between feature prototypes), the router allocates more discretization patterns ($N_k$), providing higher capacity to learn complex boundaries.
- Low Uncertainty: For well-defined classes, fewer patterns are used, preserving stability and preventing unnecessary parameter updates that could lead to forgetting.
To further enhance discriminative power, a Contrastive Discretization Loss ($L_{Cont}$) is applied. This loss ensures that discretization patterns for the same class are pulled together while patterns for different classes are pushed apart, creating a more robust feature space.
Experimental Evaluation
The researchers evaluated S6MOD across three standard OCL benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet. These tests were conducted across various memory buffer sizes (from $M=500$ to $M=10,000$) to assess efficiency under different storage constraints.
Key Results
S6MOD was tested as an enhancement to seven state-of-the-art OCL methods, including ER, OCM, and OnPro. The results were consistently positive:
- Performance Gains: The module provided a significant accuracy boost across all methods. For instance, it improved the OnPro method by approximately 4.2% on CIFAR-100 and 4.5% on Tiny-ImageNet.
- State-of-the-Art Accuracy: When paired with top-tier methods, S6MOD achieved record-breaking results, such as 79.51% on CIFAR-10.
- Visualizing Feature Separation: The impact of S6MOD is clearly visible in feature space visualizations.
As illustrated in Figure 6, the t-SNE plots show that S6MOD helps the model produce much cleaner class separations compared to the base methods alone. This improved clustering directly correlates with the reduced forgetting rates observed in the quantitative results.
Discussion and Future Work
S6MOD demonstrates that treating the discretization parameter as a learnable mixture is a powerful way to handle multi-scale temporal features in streaming data. By intelligently routing resources to "difficult" classes, the model mimics a more human-like focus on challenging information.
However, some limitations remain. The addition of an auxiliary MoE branch increases computational overhead (FLOPs) and parameter count. Furthermore, the reliance on class prototypes could present a memory bottleneck in systems with thousands of classes. Future work may focus on optimizing the routing efficiency or extending the "Mixture of Discretization" concept to other domains like video-based learning or complex time-series analysis.
Summary updated