Learning with Style: Continual Semantic Segmentation Across Tasks and Domains

Marco Toldo, Umberto Michieli, Pietro Zanuttigh

Abstract: Deep learning models dealing with image understanding in real-world settings must be able to adapt to a wide variety of tasks across different domains. Domain adaptation and class incremental learning deal with domain and task variability separately, whereas their unified solution is still an open problem. We tackle both facets of the problem together, taking into account the semantic shift within both input and label spaces. We start by formally introducing continual learning under task and domain shift. Then, we address the proposed setup by using style transfer techniques to extend knowledge across domains when learning incremental tasks and a robust distillation framework to effectively recollect task knowledge under incremental domain shift. The devised framework (LwS, Learning with Style) is able to generalize incrementally acquired task knowledge across all the domains encountered, proving to be robust against catastrophic forgetting. Extensive experimental evaluation on multiple autonomous driving datasets shows how the proposed method outperforms existing approaches, which prove to be ill-equipped to deal with continual semantic segmentation under both task and domain shift.

The Challenge of Dual Incremental Learning

Deep learning models are notoriously brittle when faced with change. In the context of semantic segmentation—the task of labeling every pixel in an image—two primary challenges exist: Class Incremental Learning (CIL), which involves learning new categories over time, and Domain Adaptation (DA), which involves adapting to new environments (e.g., different lighting, weather, or geography).

While researchers have traditionally tackled these problems separately, real-world agents like autonomous vehicles face both simultaneously. A robot might arrive in a new city (domain shift) and suddenly need to recognize a new type of delivery vehicle (task shift). Existing methods struggle in this "double whammy" scenario because the "background" class in one step might contain the "foreground" objects of the next, leading to massive confusion and catastrophic forgetting.

Conceptual overview of the LwS framework tackling simultaneous task and domain shifts.

Methodology: Learning with Style (LwS)

The core innovation of the Learning with Style (LwS) framework is the use of Domain Stylization to bridge the gap between past and present knowledge without the need to store old, privacy-sensitive image data.

Fourier-based Style Transfer

Instead of using computationally heavy Generative Adversarial Networks (GANs), LwS employs a lightweight Fourier Transform approach. The authors observe that in the frequency domain, the amplitude component captures the "style" (lighting, texture), while the phase component captures the "content" (object shapes, layout). By swapping amplitudes, the model can "re-style" current images to look like previous environments while keeping the semantic content perfectly aligned with current labels.

The Style Bank and Four-Way Training Objective

The model maintains a "Style Bank"—a collection of compact style tensors (average low-frequency amplitudes) for every domain encountered. At each learning step, the model optimizes a four-way objective to ensure comprehensive knowledge retention and adaptation:

The LwS training pipeline, featuring Domain Stylization and the four-way training objective.
  1. New Classes / New Domain: Standard supervised learning on current data.
  2. New Classes / Old Domains: Current images are stylized with "old styles" from the bank to teach the model how new classes appear in previous environments, encouraging generalization.
  3. Old Classes / New Domain: The model uses "pseudo-labels" from its previous iteration to learn how old classes look in the new environment.
  4. Old Classes / Old Domains: Knowledge distillation on stylized images ensures the model retains its ability to segment old classes in their original environments.

Experimental Evaluation

The authors evaluated LwS on a rigorous suite of autonomous driving benchmarks, including sequences of Cityscapes → BDD100K → IDD, the Mapillary Vistas dataset (split by continents), and the SHIFT synthetic dataset (simulating day-to-night transitions).

Key Results

  • Superior Stability: LwS outperformed state-of-the-art incremental learning methods (like MiB and PLOP) by a wide margin. On the Cityscapes sequence, LwS reduced the performance gap to an "Oracle" model (trained on all data at once) to ~31%, compared to ~43-44% for competitors.
  • Forward & Backward Transfer: LwS demonstrated a unique ability to improve performance on past domains even after moving to new ones (backward transfer) and performed better on unseen domains (forward transfer) than its peers.
  • Privacy-Friendly Rehearsal: By using style as a proxy for memory, LwS avoids the storage of raw images, making it more suitable for privacy-sensitive applications.

Conclusion and Future Work

LwS proves that storing a compact "style" signature is sufficient to effectively "hallucinate" old environments and prevent forgetting in complex, shifting scenarios. While the current work assumes discrete learning steps, future research will move toward continuous, stream-based learning where domain and task shifts are blurry and ever-changing.

Summary updated

Abstract

Deep learning models dealing with image understanding in real-world settings must be able to adapt to a wide variety of tasks across different domains. Domain adaptation and class incremental learning deal with domain and task variability separately, whereas their unified solution is still an open problem. We tackle both facets of the problem together, taking into account the semantic shift within both input and label spaces. We start by formally introducing continual learning under task and domain shift. Then, we address the proposed setup by using style transfer techniques to extend knowledge across domains when learning incremental tasks and a robust distillation framework to effectively recollect task knowledge under incremental domain shift. The devised framework (LwS, Learning with Style) is able to generalize incrementally acquired task knowledge across all the domains encountered, proving to be robust against catastrophic forgetting. Extensive experimental evaluation on multiple autonomous driving datasets shows how the proposed method outperforms existing approaches, which prove to be ill-equipped to deal with continual semantic segmentation under both task and domain shift.
BibTeX
@article{toldo2024learning,
  author = {Marco Toldo and Umberto Michieli and Pietro Zanuttigh},
  title = {{Learning with Style: Continual Semantic Segmentation Across Tasks and Domains}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  url = {https://openreview.net/forum?id=ykTVg8LY28},
}
Figure