An EM Framework for Online Incremental Learning of Semantic Segmentation

Shipeng Yan, Jiale Zhou, Jiangwei Xie, Songyang Zhang, Xuming He

Abstract

Online incremental learning for semantic segmentation aims to learn new visual categories from a continuous data stream while preserving knowledge of previously seen classes. This task is challenging due to catastrophic forgetting, background concept drift, and strict memory constraints. We present a unified framework based on Expectation-Maximization (EM) that treats missing labels of old classes as latent variables. By iteratively relabeling the background (E-step) and updating the model (M-step), our approach effectively mitigates forgetting and handles concept drift. Experiments on PASCAL VOC 2012 and ADE20K demonstrate that our method significantly outperforms existing state-of-the-art incremental learning techniques in online settings.

Introduction

Most deep learning models for semantic segmentation operate under a "closed-world" assumption, where they are trained once on a fixed set of categories. In real-world streaming environments, models must adapt to new classes on the fly. This Online Incremental Learning setting presents three major hurdles:

  1. Catastrophic Forgetting: Updating model weights for new knowledge often leads to the abrupt loss of previously learned information.
  2. Background Concept Drift: Pixels belonging to future classes are often labeled as "background" in early tasks, leading the model to learn incorrect associations.
  3. Online Constraints: Models must learn from a single pass of data with a very limited memory buffer for past examples.

Methodology

The proposed framework addresses these challenges using a unified strategy based on the Expectation-Maximization (EM) algorithm.

The EM Framework

The framework treats the unknown labels of old classes in new images as latent variables.

  • The E-Step (Iterative Relabeling): Since incoming data only labels new classes, the model uses its current state to predict labels for unlabeled pixels. This "relabeling" prevents the model from incorrectly learning that old classes belong to the background.
  • The M-Step (Model Update): The model parameters are updated using a composite loss function that balances ground-truth labels for new classes with the pseudo-labels generated in the E-step.
The proposed EM framework for online incremental learning, illustrating the E-step (iterative relabeling) and M-step (model update).

Memory and Optimization

To further enhance performance under online constraints, the authors introduce:

  • Dynamic Sampling: A Gibbs distribution-based sampling method selects classes the model is currently struggling with, ensuring focused learning on low-confidence areas.
  • Class-Balanced Exemplar Selection (CBES): A specialized memory management technique that maintains a diverse representation of all seen classes within a small buffer.
  • Cosine Normalization: Replacing standard linear layers with cosine distance-based weights reduces bias toward classes with higher data frequency.

Experimental Results

The framework was evaluated on the PASCAL VOC 2012 (21 classes) and ADE20K (151 classes) benchmarks.

Quantitative Performance

The proposed method consistently outperformed state-of-the-art baselines such as MiB and iCaRL. In the PASCAL VOC "15-1" disjoint task, the model achieved a 65.14% mIoU, significantly higher than the 59.78% achieved by the best baseline.

Quantitative comparison of mIoU across different incremental learning scenarios on PASCAL VOC 2012 and ADE20K benchmarks.

Robustness and Efficiency

The model demonstrated remarkable robustness to memory constraints. Even with a buffer of only 20 images, it maintained high performance while other methods collapsed. Qualitatively, the EM framework preserves crisp segmentation boundaries for both old and new classes, whereas competing methods often "erase" old knowledge by misclassifying it as background.

Conclusion

By treating the background as a collection of latent variables rather than a void, this EM-based framework provides a principled solution to concept drift and forgetting in online semantic segmentation. While the iterative E-step introduces some computational overhead, the gains in stability and plasticity make it a powerful approach for real-world streaming applications. Future work may explore memory-less variants or extensions to video segmentation.

Summary updated

BibTeX
@inproceedings{yan2021em,
  author = {Shipeng Yan and Jiale Zhou and Jiangwei Xie and Songyang Zhang and Xuming He},
  title = {{An EM Framework for Online Incremental Learning of Semantic Segmentation}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=kHcsEVXeRG},
}
Figure