Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection

Hyewon Park, Hyejin Park, Jueun Ko, Dongbo Min

Abstract

Continual Test Time Adaptation (CTTA) has emerged as a critical approach for bridging the domain gap between controlled training environments and real-world scenarios, enhancing model adaptability and robustness. Existing CTTA methods, typically categorized into Full-Tuning (FT) and Efficient-Tuning (ET), struggle with effectively addressing domain shifts. To overcome these challenges, we propose Hybrid-TTA, a holistic approach that dynamically selects an instance-wise tuning method for optimal adaptation. Our approach introduces the Dynamic Domain Shift Detection (DDSD) strategy, which identifies domain shifts by leveraging temporal correlations in input sequences and dynamically switches between FT and ET to adapt to varying domain shifts effectively. Additionally, the Masked Image Modeling based Adaptation (MIMA) framework is integrated to ensure domain-agnostic robustness with minimal computational overhead. Our Hybrid-TTA achieves a notable 1.6%p improvement in mIoU on the Cityscapes-to-ACDC benchmark dataset, surpassing previous state-of-the-art methods and offering a robust solution for real-world continual adaptation challenges.

Introduction

Deep learning models deployed in real-world scenarios, such as autonomous driving, often encounter significant performance degradation due to "domain shift." This occurs when a model trained in controlled environments (e.g., clear weather) faces unpredictable conditions like fog, rain, or night. Continual Test-Time Adaptation (CTTA) addresses this by enabling models to adapt to target domains on-the-fly using unlabeled data.

However, researchers face a "stability-plasticity" dilemma. Full-Tuning (FT) updates all parameters for rapid adaptation but risks catastrophic forgetting and error accumulation from noisy labels. Conversely, Efficient-Tuning (ET) updates a small subset of parameters (e.g., Adapters), providing stability but often failing to handle major environmental shifts. Furthermore, many current CTTA methods are computationally prohibitive for real-time use, often taking several seconds to process a single frame.

Methodology: The Hybrid-TTA Framework

Hybrid-TTA addresses these issues by dynamically balancing plasticity and stability through two core components: Dynamic Domain Shift Detection (DDSD) and Masked Image Modeling based Adaptation (MIMA).

The Hybrid-TTA framework utilizes a Teacher-Student architecture. The DDSD module determines the tuning strategy (Full vs. Adapter) based on domain shift detection, while MIMA provides self-supervised regularization through image reconstruction.

Dynamic Domain Shift Detection (DDSD)

Rather than employing a static tuning strategy, Hybrid-TTA utilizes DDSD to select the optimal adaptation method for each input frame. By leveraging a Mean Teacher framework, the system monitors the discrepancy between a fast-adapting Student model and a stable, exponential moving average Teacher model. A sudden spike in prediction discrepancy signals a significant domain shift (e.g., entering a tunnel), triggering Full-Tuning to allow the model to adapt quickly. In stable environments, the system reverts to Adapter-Tuning (updating only ~0.1% of parameters) to maintain efficiency and prevent knowledge loss.

Masked Image Modeling based Adaptation (MIMA)

To mitigate the risk of model collapse from noisy pseudo-labels, Hybrid-TTA incorporates MIMA as a self-supervised auxiliary task. The model is trained to reconstruct masked portions of the input image, forcing the encoder to learn robust, domain-agnostic visual features. This reconstruction task acts as an "anchor," ensuring the model retains a fundamental understanding of scene geometry even when segmentation labels are unreliable.

Experimental Evaluation

The framework was evaluated on challenging benchmarks including Cityscapes-to-ACDC, OnDA, and OASIS. Hybrid-TTA demonstrates superior performance in both accuracy and computational efficiency.

Comparison of mIoU vs. FPS on the Cityscapes-to-ACDC benchmark. Hybrid-TTA achieves state-of-the-art accuracy while operating significantly faster than previous top-performing methods like CoTTA and SVDP.

On the Cityscapes-to-ACDC benchmark, Hybrid-TTA achieved a state-of-the-art 62.2% mIoU. Crucially, it operates at 2.004 FPS, representing a 20-fold speedup over previous leaders like CoTTA and SVDP, which typically run at ~0.1 FPS. Long-term stability tests also confirmed that Hybrid-TTA effectively resists catastrophic forgetting over multiple environmental cycles, maintaining high performance where full-tuning methods often decay.

Discussion and Conclusion

Hybrid-TTA offers a pragmatic solution to the stability-plasticity trade-off in CTTA. By treating adaptation as an instance-wise decision and anchoring learning with self-supervised reconstruction, it achieves high accuracy with near-real-time throughput.

While the 20-fold speedup is a significant step toward real-world viability, future work is needed to reach the 30+ FPS required for high-speed autonomous navigation. Additionally, extending this hybrid approach to other computer vision tasks, such as object detection or depth estimation, remains a promising direction for future research.

Summary updated

Abstract

Continual Test Time Adaptation (CTTA) has emerged as a critical approach for bridging the domain gap between the controlled training environments and the real-world scenarios, enhancing model adaptability and robustness. Existing CTTA methods, typically categorized into Full-Tuning (FT) and Efficient-Tuning (ET), struggle with effectively addressing domain shifts. To overcome these challenges, we propose Hybrid-TTA, a holistic approach that dynamically selects instance-wise tuning method for optimal adaptation. Our approach introduces the Dynamic Domain Shift Detection (DDSD) strategy, which identifies domain shifts by leveraging temporal correlations in input sequences and dynamically switches between FT and ET to adapt to varying domain shifts effectively. Additionally, the Masked Image Modeling based Adaptation (MIMA) framework is integrated to ensure domain-agnostic robustness with minimal computational overhead. Our Hybrid-TTA achieves a notable 1.6%p improvement in mIoU on the Cityscapes-to-ACDC benchmark dataset, surpassing previous state-of-the-art methods and offering a robust solution for real-world continual adaptation challenges.
BibTeX
@inproceedings{park2025hybrid,
  author = {Hyewon Park and Hyejin Park and Jueun Ko and Dongbo Min},
  title = {{Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=lirYEX2KQQ},
}
Figure