Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL

Shibin Su, Guoqiang Liang, De Cheng, Shizhou Zhang, Lingyan Ran, Yanning ZhangarXiv 2025 · DOI · arXiv

Abstract

Online Class-Incremental Learning (OCIL) requires models to learn from a continuous data stream where each sample is seen only once, often leading to catastrophic forgetting. This paper introduces a novel framework that addresses the stability-plasticity dilemma by combining ensemble learning with a Global Workspace Model (GWM). The GWM acts as a shared, implicit memory that anchors multiple "student" models through multi-level collaborative distillation and periodic parameter fusion. Our approach significantly outperforms current state-of-the-art methods on benchmarks such as CIFAR-100 and Tiny-ImageNet, providing a more robust balance between learning new information and retaining historical knowledge.

1. Introduction

In real-world applications, AI systems must adapt to evolving data distributions. Class-Incremental Learning (CIL) focuses on learning new classes over time. The "Online" variant (OCIL) is particularly demanding, as it restricts training to a single epoch and imposes severe memory constraints on the replay buffer. This environment intensifies the stability-plasticity dilemma: the model must be plastic enough to learn new classes rapidly but stable enough to prevent the erasure of previously acquired knowledge.

While ensemble methods have shown promise in enhancing plasticity, they often suffer from divergence among individual models, leading to inconsistent representations of old tasks. We propose a unified framework that utilizes a Global Workspace Model to synchronize an ensemble of learners, ensuring a cohesive and stable knowledge base.

2. Methodology: The GWM Framework

The proposed architecture consists of two student models and a central Global Workspace Model (GWM), as illustrated in Figure 1.

Overview of the proposed framework featuring multi-level collaborative distillation and the Global Workspace Model.

2.1 The Global Workspace Model (GWM)

Inspired by cognitive science, the GWM serves as a dynamic anchor representing the ensemble's historical learning trajectory. It is constructed as a linear combination of the students' parameters, updated via an Exponential Moving Average (EMA). This mechanism encourages the model to converge toward "flat" loss landscapes, which are associated with superior generalization and robustness against forgetting.

2.2 Multi-level Collaborative Distillation (MCD)

To maintain consistency and leverage the GWM, we implement three layers of knowledge distillation:

  1. Peer-to-Peer Distillation: Students align their predictions with one another to foster diversity and improve collective plasticity.
  2. Student-to-GWM Distillation: Each student is anchored to the GWM's predictions, ensuring they do not drift too far from the established historical consensus.
  3. Cross-View Consistency: Students are trained to produce consistent outputs for different augmented versions of the same input, further stabilizing feature representations.

2.3 Periodic Parameter Fusion

To prevent divergence in the high-dimensional parameter space, the GWM’s weights are periodically "pushed" back into the student models. This fusion step ensures that the students remain within the same functional region, mitigating feature drift as new tasks are introduced.

3. Experimental Evaluation

We evaluated our framework on three standard OCIL benchmarks:

  • Split CIFAR-100: 10 tasks with 10 classes each.
  • Split Tiny-ImageNet: 100 tasks with 2 classes each, representing a high-frequency task-shift scenario.
  • Split ImageNet-100: 10 tasks with 10 classes each.

Experiments were conducted under various memory budgets for the replay buffer to test the framework's efficiency in resource-constrained settings.

4. Results and Discussion

Our method consistently achieves superior performance compared to established baselines, including Experience Replay (ER), Supervised Contrastive Replay (SCR), and recent state-of-the-art methods like CCL-DC.

As shown in Figure 4, the proposed framework maintains higher Average Accuracy (AA) throughout the incremental learning process. For instance, on CIFAR-100 with a 1K memory budget, our method improved the Final Average Accuracy (FAA) of the ER baseline from 14.3% to 38.4%. Notably, the GWM approach outperformed the previous state-of-the-art (CCL-DC) by approximately 10% to 13% on the same dataset.

Beyond accuracy, the framework demonstrates high efficiency. Despite the use of multiple models, it requires less running time and GPU memory than some complex single-model ensemble alternatives like OCM. Visualization of the stability-plasticity trade-off confirms that our method resides closer to the ideal Pareto front than competing approaches.

5. Conclusion

We have presented a unified framework for OCIL that leverages a Global Workspace Model and multi-level collaborative distillation to effectively manage the stability-plasticity dilemma. By anchoring an ensemble of students to a shared historical consensus, we significantly mitigate catastrophic forgetting while maintaining high plasticity. Future work will explore the use of heterogeneous architectures within the workspace and more advanced data augmentation strategies to further enhance the system's robustness.

Summary updated

Abstract

Online Class-Incremental Learning (OCIL) enables models to learn continuously from non-i.i.d. data streams. Since samples of the data streams can be seen only once, it is more suitable for real-world scenarios compared to offline learning. However, this constraint intensifies the challenge for OCIL in maintaining an appropriate balance between stability and plasticity. Moreover, under stricter memory buffer constraints in real world, current replay-based methods are less effective. While ensemble methods improve plasticity, they often struggle with stability. Inspired by the Global Workspace Theory (GWT), we propose a novel approach that enhances ensemble learning through a Global Workspace Model (GWM)-a shared, implicit memory that guides the learning of multiple student models. The GWM is formed by fusing the parameters of all students within each training batch, capturing the historical learning trajectory and serving as a dynamic anchor for knowledge consolidation. Like the broadcasting mechanism of GWT, the GWM is redistributed periodically to students, stabilizing learning and promoting cross-task consistency. In addition, we introduce a multi-level collaborative distillation mechanism. It enforces peer-to-peer consistency among students and preserves historical knowledge by aligning each student with the GWM. As a result, student models remain adaptable to new tasks while maintaining previously learned knowledge, striking a better balance between stability and plasticity. Extensive experiments on three standard OCIL benchmarks show that our method delivers significant performance improvement for several OCIL models across various memory budgets. The code is available at https://github.com/susususushi/GWM.
BibTeX
@misc{su2025multi,
  author = {Shibin Su and Guoqiang Liang and De Cheng and Shizhou Zhang and Lingyan Ran and Yanning Zhang},
  title = {{Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL}},
  year = {2025},
  eprint = {2508.08677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.08677},
}
Figure