Endpoints Weight Fusion for Class Incremental Semantic Segmentation

Jia-Wen Xiao, Chang-Bin Zhang, Jiekang Feng, Xialei Liu, Joost van de Weijer, Ming-Ming ChengCVPR 2023 · DOI

Abstract

Class-incremental semantic segmentation (CISS) is plagued by catastrophic forgetting, where updating a model with new classes leads to a sharp decline in performance on previously learned ones. This paper introduces Endpoints Weight Fusion (EWF), a straightforward yet powerful method to mitigate this issue. By performing a dynamic linear interpolation of model weights between the start and end of an incremental training step, EWF effectively balances plasticity and stability. Experimental results on PASCAL VOC and ADE20K demonstrate that EWF sets new state-of-the-art performance levels, particularly in long-sequence incremental scenarios, with zero additional training overhead.

Introduction

The goal of Class Incremental Semantic Segmentation (CISS) is to enable models to learn new object categories sequentially. A major hurdle is catastrophic forgetting, exacerbated in segmentation by the "semantic shift" where old classes appear as unlabeled background in new training batches. While existing methods rely heavily on knowledge distillation to align output representations, they often fail to preserve the underlying parameter-level knowledge effectively.

Methodology

Endpoints Weight Fusion (EWF) proposes that the optimal model for an incremental task lies on the interpolation path between the "starting point" (the model before the current step) and the "ending point" (the model after training on new data).

The balanced model weights $\theta_{balanced}$ are calculated as: $$\theta_{balanced} = \alpha_t \theta_{new} + (1 - \alpha_t) \theta_{old}$$

The dynamic fusion factor $\alpha_t$ is determined by the ratio of new classes to the total number of classes: $\alpha_t = \sqrt{\frac{N_{new}}{N_{new} + N_{old}}}$. This adaptive scaling ensures the model retains sufficient knowledge of the expanding class set. To ensure the weights remain in a region of the parameter space where linear fusion is viable, standard distillation losses are employed during the training of the new model.

Figure 2: Overview of the EWF framework, illustrating the training and test pipeline where 'old' and 'new' model weights are fused into a 'balanced' model.

Unlike Exponential Moving Average (EMA), which aggregates weights throughout the training process, EWF performs fusion only after the model's representation has stabilized. This avoids the influence of noisy gradients early in the incremental training phase.

Experiments and Results

The effectiveness of EWF was validated on PASCAL VOC 2012 and the large-scale ADE20K dataset. In the PASCAL VOC 10-1 setting, which involves 11 incremental steps, EWF improved the mIoU of the MiB baseline by over 24% and the PLOP baseline by over 21%.

Figure 4: Performance comparison (mIoU) over incremental steps on PASCAL VOC for the 10-1 and 15-1 scenarios, showing EWF's superior stability.

On the ADE20K 100-5 configuration, EWF achieved a state-of-the-art mIoU of 32.1%. These results highlight EWF's ability to maintain high performance over long sequences of updates, a common failure point for previous methods.

Discussion and Conclusion

A key insight of this work is the "collapse and recovery" phenomenon: during incremental training, the model's similarity to its previous state initially drops before recovering. EWF capitalizes on this by fusing weights at the "endpoint" of this recovery. Furthermore, because the fusion happens post-training, it adds no computational cost to the training pipeline.

While EWF is highly effective, its performance is contingent on the proximity of the two endpoint models in parameter space, necessitating the continued use of distillation as a regularizer. Future research will explore the applicability of weight fusion to other incremental tasks such as object detection and instance segmentation. In summary, EWF provides a simple, efficient, and robust solution to catastrophic forgetting in CISS by shifting the focus from output-space constraints to parameter-space fusion.

Summary updated

BibTeX
@inproceedings{xiao2023endpoints,
  author = {Jia-Wen Xiao and Chang-Bin Zhang and Jiekang Feng and Xialei Liu and Joost van de Weijer and Ming-Ming Cheng},
  title = {{Endpoints Weight Fusion for Class Incremental Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00696},
}
Figure