Abstract
Continual Test Time Adaptation (CTTA) has emerged as a critical approach for bridging the domain gap between controlled training environments and real-world scenarios, enhancing model adaptability and robustness. Existing CTTA methods, typically categorized into Full-Tuning (FT) and Efficient-Tuning (ET), struggle with effectively addressing domain shifts. To overcome these challenges, we propose Hybrid-TTA, a holistic approach that dynamically selects an instance-wise tuning method for optimal adaptation. Our approach introduces the Dynamic Domain Shift Detection (DDSD) strategy, which identifies domain shifts by leveraging temporal correlations in input sequences and dynamically switches between FT and ET to adapt to varying domain shifts effectively. Additionally, the Masked Image Modeling based Adaptation (MIMA) framework is integrated to ensure domain-agnostic robustness with minimal computational overhead. Our Hybrid-TTA achieves a notable 1.6%p improvement in mIoU on the Cityscapes-to-ACDC benchmark dataset, surpassing previous state-of-the-art methods and offering a robust solution for real-world continual adaptation challenges.
Introduction
Deep learning models deployed in real-world scenarios, such as autonomous driving, often encounter significant performance degradation due to "domain shift." This occurs when a model trained in controlled environments (e.g., clear weather) faces unpredictable conditions like fog, rain, or night. Continual Test-Time Adaptation (CTTA) addresses this by enabling models to adapt to target domains on-the-fly using unlabeled data.
However, researchers face a "stability-plasticity" dilemma. Full-Tuning (FT) updates all parameters for rapid adaptation but risks catastrophic forgetting and error accumulation from noisy labels. Conversely, Efficient-Tuning (ET) updates a small subset of parameters (e.g., Adapters), providing stability but often failing to handle major environmental shifts. Furthermore, many current CTTA methods are computationally prohibitive for real-time use, often taking several seconds to process a single frame.
Methodology: The Hybrid-TTA Framework
Hybrid-TTA addresses these issues by dynamically balancing plasticity and stability through two core components: Dynamic Domain Shift Detection (DDSD) and Masked Image Modeling based Adaptation (MIMA).
Dynamic Domain Shift Detection (DDSD)
Rather than employing a static tuning strategy, Hybrid-TTA utilizes DDSD to select the optimal adaptation method for each input frame. By leveraging a Mean Teacher framework, the system monitors the discrepancy between a fast-adapting Student model and a stable, exponential moving average Teacher model. A sudden spike in prediction discrepancy signals a significant domain shift (e.g., entering a tunnel), triggering Full-Tuning to allow the model to adapt quickly. In stable environments, the system reverts to Adapter-Tuning (updating only ~0.1% of parameters) to maintain efficiency and prevent knowledge loss.
Masked Image Modeling based Adaptation (MIMA)
To mitigate the risk of model collapse from noisy pseudo-labels, Hybrid-TTA incorporates MIMA as a self-supervised auxiliary task. The model is trained to reconstruct masked portions of the input image, forcing the encoder to learn robust, domain-agnostic visual features. This reconstruction task acts as an "anchor," ensuring the model retains a fundamental understanding of scene geometry even when segmentation labels are unreliable.
Experimental Evaluation
The framework was evaluated on challenging benchmarks including Cityscapes-to-ACDC, OnDA, and OASIS. Hybrid-TTA demonstrates superior performance in both accuracy and computational efficiency.
On the Cityscapes-to-ACDC benchmark, Hybrid-TTA achieved a state-of-the-art 62.2% mIoU. Crucially, it operates at 2.004 FPS, representing a 20-fold speedup over previous leaders like CoTTA and SVDP, which typically run at ~0.1 FPS. Long-term stability tests also confirmed that Hybrid-TTA effectively resists catastrophic forgetting over multiple environmental cycles, maintaining high performance where full-tuning methods often decay.
Discussion and Conclusion
Hybrid-TTA offers a pragmatic solution to the stability-plasticity trade-off in CTTA. By treating adaptation as an instance-wise decision and anchoring learning with self-supervised reconstruction, it achieves high accuracy with near-real-time throughput.
While the 20-fold speedup is a significant step toward real-world viability, future work is needed to reach the 30+ FPS required for high-speed autonomous navigation. Additionally, extending this hybrid approach to other computer vision tasks, such as object detection or depth estimation, remains a promising direction for future research.
Summary updated