Towards Domain-Aware Knowledge Distillation for Continual Model Generalization

Nikhil Reddy, Mahsa Baktashmotlagh, Chetan AroraWACV 2024

TL;DR

This paper introduces "DoSe," a novel framework for continual model generalization in autonomous navigation systems. DoSe tackles the challenge of catastrophic forgetting in changing environments by using domain-aware batch normalization prototypes for self-distillation and an exemplar-based replay buffer to retain knowledge from previously seen domains.

What Problem Does This Paper Address?

Autonomous navigation systems, like self-driving cars, rely heavily on Deep Neural Networks (DNNs) to interpret their surroundings. A critical requirement for these systems is generalization – the ability to perform well even when encountering new, unseen environments or conditions. However, DNNs often suffer from "catastrophic forgetting," where learning new information causes them to forget previously acquired knowledge. This is particularly problematic in real-world scenarios like autonomous driving, where environmental conditions (weather, lighting) can change drastically and continuously. Existing approaches like Domain Generalization (DG) and Unsupervised Domain Adaptation (UDA) struggle with these dynamic shifts and often require access to source data, which can be impractical due to storage and privacy constraints. Test-Time Adaptation (TTA) methods attempt to adapt models during inference but often falter in continually changing environments. This paper addresses the need for a method that can adapt to evolving target domains in an online, continuous manner without access to source data, while effectively mitigating catastrophic forgetting.

Methods

The core of the DoSe framework lies in two main innovations:

  • Domain-Aware Knowledge Self-Distillation: DoSe leverages batch normalization (BN) statistics, which are sensitive to domain shifts. It enforces consistency between the BN prototypes of a "teacher" model (the model from the previous iteration) and a "student" model (the current model being updated) for batches of images from the same target domain. This self-distillation process, guided by these domain-aware BN prototypes, helps the student model learn to mimic the teacher's understanding of the current domain.
  • Exemplar-Based Replay Buffer: To combat catastrophic forgetting, DoSe employs an exemplar-based replay buffer. This buffer stores "difficult" samples identified using a metric called Variance of Gradients (VoG). VoG helps pinpoint samples that cause significant changes in model gradients, indicating they are challenging for the model to predict. These difficult samples are periodically replayed to the model, reinforcing knowledge learned from earlier domains and preventing it from being overwritten.

The overall training process involves adapting a pre-trained model to incoming target data in an online fashion. The student model is trained using a combination of cross-entropy loss on pseudo-labels generated by the teacher model and the proposed domain-preserving similarity distillation loss.

Datasets

The authors evaluated their framework on the following datasets:

  • ACDC: Contains 4006 images of adverse weather conditions (fog, dark, snow, rain) for urban-scene semantic segmentation.
  • C-Driving: A dataset with four adverse weather conditions (cloudy, rainy, snowy, overcast) collected from BDD100K.
  • SHIFT: A large synthetic dataset with 2.5 million images, used with fog, rain, night, and overcast conditions.
  • Cityscapes (Day): A standard dataset for urban-scene semantic segmentation under clear daytime conditions, used as a source domain.

Key Insights and Ideas

  • BN Statistics as Domain Signatures: The paper highlights that BN statistics, often overlooked or simply adapted, can serve as powerful indicators of domain shifts. By enforcing consistency in these statistics between teacher and student models, DoSe effectively distills domain-specific knowledge.
  • Targeted Replay for Forgetting Mitigation: Instead of randomly replaying data, using a metric like VoG to identify and replay difficult samples is a more efficient strategy for preserving knowledge and combating catastrophic forgetting. This suggests that focusing on challenging examples is key to robust continual learning.
  • Unified Framework for Multiple Challenges: DoSe demonstrates its effectiveness across various settings, including adapting from daytime to adverse conditions, improving existing Domain Generalization (DG) and Unsupervised Domain Adaptation (UDA) methods, and handling continual shifts in real-world scenarios.

Key Results

  • Significant Performance Gains: On the ACDC dataset, DoSe outperformed existing state-of-the-art methods in Domain Generalization (26% improvement), Unsupervised Domain Adaptation (14% improvement), and Daytime settings (70% improvement).
  • Improved Generalization: When applied to existing SOTA daytime models, DoSe showed substantial improvements in mIoU across various adverse weather conditions. For instance, it improved DeepLabv3+ ResNet101 by 8% on the Cityscapes-val dataset.
  • Effective Forgetting Mitigation: DoSe demonstrated its ability to alleviate catastrophic forgetting, outperforming methods that do not explicitly address this issue.
  • Faster Adaptation: Compared to COTTA, DoSe showed a faster inference time for adapting to new domains, making it more suitable for real-time applications.

Limitations and Future Work

  • Computational Overhead: While DoSe aims for efficiency, the paper notes that methods like COTTA have significantly higher inference times. DoSe's inference time is presented as an improvement, but it's still a factor to consider.
  • Order of Sequence: The paper mentions an ablation study to understand the effect of the order of domain sequences, suggesting that this could be a factor influencing performance.
  • Future Work: The paper implicitly suggests that further exploration into more sophisticated methods for identifying difficult samples or more efficient distillation techniques could lead to even better performance. The framework's success across various datasets and tasks also opens avenues for applying it to other domains beyond autonomous navigation.

Summary updated

BibTeX
@inproceedings{reddy2024towards,
  author = {Nikhil Reddy and Mahsa Baktashmotlagh and Chetan Arora},
  title = {{Towards Domain-Aware Knowledge Distillation for Continual Model Generalization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure