TL;DR
This paper introduces "Multi-Domain Incremental Learning for Semantic Segmentation" (MDIL-SS), a novel framework designed to enable semantic segmentation models to learn new geographic driving domains sequentially without forgetting previously acquired knowledge. The key innovation lies in a dynamic architecture that balances shared, domain-invariant parameters with domain-specific ones, coupled with a sophisticated optimization strategy to achieve both stability and plasticity.
What Problem Does This Paper Address?
Autonomous driving systems rely heavily on semantic segmentation to understand their surroundings. However, real-world driving environments vary significantly across geographies due to differences in road conditions, weather, lighting, and even the presence of specific objects (e.g., auto-rickshaws in India vs. cars in the US). Training a single, universal model on all data simultaneously is often impractical due to data privacy, storage limitations, and the sheer scale of data. A more realistic approach is incremental learning, where a model learns new domains sequentially. The core challenge, known as "catastrophic forgetting," is that when a model is fine-tuned on a new domain, it tends to forget what it learned from previous domains. Existing methods struggle with this, especially when dealing with visually disparate geographical domains and potentially non-overlapping label sets. This paper tackles this problem by proposing a method for multi-domain incremental semantic segmentation that explicitly addresses both domain shift and label space variations without requiring access to old data.
Methods
The proposed framework, MDIL-SS, employs a dynamic architecture that splits the network's parameters into two categories:
- Domain-Invariant Parameters (Shared): These parameters, forming the core of the encoder, are designed to capture universal semantic features common across all geographical domains.
- Domain-Specific Parameters: These are dedicated parameters (in the decoder and adapter layers) that learn the unique characteristics of each specific domain.
This is achieved by modifying standard residual units in the encoder into Domain-Aware Residual Units (DAUs). DAUs incorporate domain-specific parallel residual adapters (DS-RAP) and domain-specific batch normalization layers (DS-BN).
The optimization strategy is crucial for balancing stability (retaining old knowledge) and plasticity (acquiring new knowledge):
- Domain-Specific Optimization: Parameters for a new domain are trained using a standard cross-entropy loss (LCE) on that domain's data. Previous domain-specific parameters are frozen.
- Domain-Invariant Optimization: The shared encoder weights are optimized using a combination of the current domain's loss (LCE) and a novel regularization loss, KL-divergence (KLD) loss (LKLD). This LKLD loss acts as a form of "knowledge distillation" from the previous model's predictions to the current model's predictions on the previous domain's data, effectively preventing the shared weights from drifting too far from previously learned representations.
- Differential Learning Rates (dlr): A significantly lower learning rate is applied to the shared weights (Ws) compared to the domain-specific weights (Wt). This helps stabilize the shared parameters and prevent them from being overly influenced by the new domain's data.
- Initialization Strategy (initw): Domain-specific parameters are initialized from their counterparts in the previous step, rather than random initialization, which is found to be detrimental.
Datasets
The paper uses three large-scale, diverse urban driving datasets for training and evaluation:
- Cityscapes (CS): Collected from urban streets in 50 European cities, featuring 19 labels.
- Berkeley Deep Drive (BDD100k): A diverse dataset from the United States, covering various weather and times of day, with a common label space.
- Indian Driving Dataset (IDD): Collected from Indian cities, characterized by unconstrained road environments and a larger label space (26 labels), including unique classes like "auto-rickshaw."
The experiments explore scenarios with both aligned and non-overlapping label spaces between these datasets.
Key Insights and Ideas
- Catastrophic forgetting is a major hurdle for semantic segmentation in multi-domain incremental learning: Simple fine-tuning drastically degrades performance on previously learned domains, highlighting the need for specialized methods.
- A dynamic architecture with parameter isolation is key: Separating domain-invariant and domain-specific parameters allows for targeted learning and retention.
- Knowledge distillation via KL-divergence is effective for preserving shared knowledge: The LKLD loss helps the shared encoder retain information from past domains by comparing its predictions to those of the previous model.
- Differential learning rates are crucial for stability-plasticity trade-off: A much lower learning rate for shared weights prevents them from "forgetting" past domains too quickly.
- Initialization matters: Randomly initializing domain-specific weights is detrimental; initializing from previous weights is a better strategy.
- The model generalizes well across different domain ordering and label space scenarios: The proposed method shows robustness whether label spaces are aligned or have significant overlaps/differences.
Key Results
- The proposed MDIL-SS model significantly mitigates catastrophic forgetting. For instance, in the CS → BDD setting, forgetting on CS is reduced by 25.16% compared to the fine-tuning baseline.
- The model achieves a good stability-plasticity trade-off, indicated by a low "Am%" score (average percentage drop in mIoU across tasks), which quantifies overall IL performance. For CS → BDD → IDD, the Am% is 10.39%, significantly better than fine-tuning.
- In the CS → BDD scenario, the model even surpasses the single-task performance on BDD by 1.63%, demonstrating positive forward transfer.
- Qualitative results (t-SNE visualizations) show that the proposed method preserves the latent space representation of learned features from older domains, unlike standard fine-tuning which distorts them. This allows for better discrimination of classes from the old domain.
- The model performs well even with non-overlapping label spaces, such as in the CS → IDD scenario, where forgetting on CS is mitigated by 27.77%.
Limitations and Future Work
- Computational Overhead: While the paper focuses on parameter sharing, the dynamic architecture and multiple domain-specific paths inherently add complexity compared to a single-task model.
- Domain Interference: The paper notes that label misalignment can lead to some domain interference on the new domain, although the method still provides promising performance. Future work could explore more advanced techniques to further mitigate this.
- Generalizability to other tasks: The current work focuses on semantic segmentation. Extending this MDIL framework to other computer vision tasks like object detection or instance segmentation would be a valuable future direction.
- Real-time performance: While ERFNet is used as a backbone, further optimization for real-time autonomous driving applications might be necessary.
Summary updated