Learning With Style: Continual Semantic Segmentation Across Tasks and Domains

Marco Toldo, Umberto Michieli, Pietro Zanuttigh

This paper, titled "Learning with Style: Continual Semantic Segmentation Across Tasks and Domains," addresses a critical hurdle in deploying computer vision for robotics and autonomous driving: the "double whammy" of learning new objects while moving through new environments.

TL;DR

The authors introduce LwS (Learning with Style), a framework that enables semantic segmentation models to learn new classes and adapt to new domains simultaneously without "catastrophic forgetting." By using Fourier-based style transfer to "replay" the look of previous environments, the model maintains high performance on old tasks while mastering new ones in diverse global settings.

What Problem Does This Paper Address?

Deep learning models are notoriously brittle when faced with change. In the context of semantic segmentation (labeling every pixel in an image), two main challenges exist:

  1. Class Incremental Learning (CIL): Learning new categories over time (e.g., first learning "road," then later learning "bicycle") usually causes the model to forget the first category.
  2. Domain Adaptation (DA): A model trained in the sunny streets of Germany (Cityscapes) often fails when deployed in the chaotic traffic of India (IDD) due to differences in lighting, weather, and architecture.

While researchers have tackled these separately, real-world agents face both at once. A robot might arrive in a new city (domain shift) and suddenly need to recognize a new type of delivery vehicle (task shift). Existing methods struggle because the "background" class in one step might contain the "foreground" objects of the next, leading to massive confusion and forgetting.

Methods

The core innovation of LwS is the use of Domain Stylization to bridge the gap between past and present knowledge without needing to store old, privacy-sensitive image data.

  • Fourier-based Style Transfer: Instead of using heavy Generative Adversarial Networks (GANs), the authors use a lightweight Fourier Transform approach. They observe that in the frequency domain, the amplitude component captures the "style" (lighting, texture), while the phase component captures the "content" (object shapes, layout).
  • The Style Bank: The model stores a tiny "style tensor" (the average low-frequency amplitude) for every domain it encounters.
  • Four-Way Training Objective: At each new learning step, the model optimizes four distinct losses:
  • New Classes/New Domain: Standard learning on current data.
  • New Classes/Old Domains: The model takes current images, applies "old styles" from the style bank, and learns the new classes on them. This encourages generalization.
  • Old Classes/New Domain: Uses "pseudo-labels" from the previous version of the model to teach the current model how old classes look in the new environment.
  • Old Classes/Old Domains: Uses knowledge distillation on stylized images to ensure the model doesn't forget how to segment old classes in their original environments.

Datasets

The authors evaluated LwS on a rigorous suite of autonomous driving benchmarks:

  • Urban Scenes: A sequence of Cityscapes (CS), BDD100K, and IDD.
  • Worldwide: The Mapillary Vistas dataset, split into six steps based on continents (Europe, North America, Asia, Oceania, Africa, South America).
  • Environmental: The SHIFT synthetic dataset, used to simulate transitions between Daytime, Twilight, and Night.

Key Insights and Ideas

  • Style as a Proxy for Memory: Traditional "rehearsal" methods require storing old images, which is memory-intensive and raises privacy concerns. LwS proves that storing a compact "style" signature is enough to "hallucinate" old environments effectively.
  • Phase is King: By keeping the phase component of the current image unchanged during stylization, the model maintains perfect alignment with its ground-truth labels while practicing on different "textures."
  • Handling the "Unknown": The authors explicitly model the "unknown" class to account for the fact that today's background might be tomorrow's new object. This prevents the model from being overconfident and erasing old knowledge.

Key Results

  • Superior Stability: On the Cityscapes $\to$ BDD $\to$ IDD sequence, LwS outperformed state-of-the-art incremental learning methods (like MiB and PLOP) by a wide margin. In terms of the relative performance gap ($\Delta$) compared to an "Oracle" (a model trained on everything at once), LwS achieved a gap of ~31%, whereas competitors lagged at ~43-44%.
  • Forward & Backward Transfer: LwS showed a unique ability to improve on past domains even after moving to new ones (backward transfer) and performed better on unseen domains (forward transfer) than its peers.
  • Robustness to Order: Whether the model learned "easy" domains first or "hard" ones, LwS remained the most stable performer.

Limitations and Future Work

  • Discrete Steps: The paper assumes the world changes in discrete "steps" (Step 1: City A, Step 2: City B). In reality, domain and task shifts are often continuous and blurry.
  • Stylization Artifacts: While Fourier stylization is efficient, it can sometimes introduce visual artifacts that don't perfectly represent real-world distribution shifts.
  • Future Direction: The authors suggest moving toward "application-oriented" settings where shifts happen in a continuous stream rather than distinct, supervised batches.

Summary updated

Abstract

Deep learning models dealing with image understanding in real-world settings must be able to adapt to a wide variety of tasks across different domains. Domain adaptation and class incremental learning deal with domain and task variability separately, whereas their unified solution is still an open problem. We tackle both facets of the problem together, taking into account the semantic shift within both input and label spaces. We start by formally introducing continual learning under task and domain shift. Then, we address the proposed setup by using style transfer techniques to extend knowledge across domains when learning incremental tasks and a robust distillation framework to effectively recollect task knowledge under incremental domain shift. The devised framework (LwS, Learning with Style) is able to generalize incrementally acquired task knowledge across all the domains encountered, proving to be robust against catastrophic forgetting. Extensive experimental evaluation on multiple autonomous driving datasets shows how the proposed method outperforms existing approaches, which prove to be ill-equipped to deal with continual semantic segmentation under both task and domain shift.
BibTeX
@article{toldo2024learning,
  author = {Marco Toldo and Umberto Michieli and Pietro Zanuttigh},
  title = {{Learning With Style: Continual Semantic Segmentation Across Tasks and Domains}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  url = {https://openreview.net/forum?id=ykTVg8LY28},
}

Loading related content...

Figure