Disentangle Source and Target Knowledge for Continual Test-Time Adaptation

Tianyi Ma, Maoying QiaoWACV 2025

TL;DR

This paper introduces SoTa-DiT, a novel method for Continual Test-Time Adaptation (CoTTA) that disentangles source and target domain knowledge using distinct visual prompts within a Vision Transformer (ViT). By preserving source knowledge and effectively extracting target knowledge separately, SoTa-DiT significantly improves image classification accuracy in continuously shifting test environments.

What Problem Does This Paper Address?

The real world is dynamic, and machine learning models often face "domain shifts" – situations where the data they encounter during testing differs from the data they were trained on. Test-Time Adaptation (TTA) aims to address this by fine-tuning models on unlabeled test data. However, standard TTA struggles with continual domain shifts, where the test data keeps changing over time. Continual Test-Time Adaptation (CoTTA) tackles this by requiring a model to adapt to a sequence of changing target domains without access to the original training data. The core challenge in CoTTA is twofold: preserving the knowledge learned from the initial source domain and simultaneously extracting new knowledge from the evolving target domains. Existing methods often struggle to balance these two objectives, either by focusing too much on one or by mixing source and target knowledge in a way that hinders performance.

Methods

SoTa-DiT leverages the Vision Transformer (ViT) architecture and introduces a "dual-prompt" mechanism. Instead of a single adaptation process, SoTa-DiT uses two distinct visual prompts:

  • Source Prompt (SP): This prompt is trained to preserve knowledge from the original source model. It uses a combination of contrastive learning and similarity loss to ensure it retains the essential source domain information. An additional "source adaptation loss" helps integrate this preserved knowledge into the rest of the model without corrupting it.
  • Target Prompt (TP): This prompt is designed to extract knowledge from the current, unlabeled target domain. It employs contrastive learning to cluster target data and a "target guiding loss" that uses predictions from both SP and TP (from a teacher model) to provide a form of pseudo-labeling, helping TP learn more accurate target-specific features.

Crucially, these prompts operate with minimal direct interaction, allowing for disentangled learning. The final prediction is an average of the outputs from the SP and TP. The overall architecture follows a teacher-student setup, where the student model is updated, and the teacher model is an exponential moving average of the student, a common practice in TTA for stability.

Datasets

The paper evaluates SoTa-DiT on three CoTTA datasets:

  • ImageNet-C: Contains 15 types of image corruptions at varying severity levels.
  • ImageNet-R: Features images with diverse renditions.
  • ImageNet-D109: Includes 109 classes with 6 types of domain shifts.

The authors also conduct ablation studies on ImageNet-C.

Key Insights and Ideas

  • Disentanglement is Key: The core idea is that separating the learning of source knowledge preservation and target knowledge extraction into distinct pathways (prompts) is more effective than trying to do both simultaneously or in a mixed fashion.
  • Prompts as Knowledge Extractors: Visual prompts in ViTs are shown to be powerful and flexible tools for guiding knowledge extraction from specific domains.
  • Balancing Act: The success of SoTa-DiT hinges on the careful design of loss functions for each prompt and the subsequent combination of their outputs.
  • ViT's Suitability: The parallel processing capabilities of ViTs make them well-suited for this prompt-based disentanglement approach.

Key Results

  • SoTa-DiT significantly outperforms state-of-the-art CoTTA methods across ImageNet-C, ImageNet-R, and ImageNet-D109, achieving improvements of up to +4.5% on ImageNet-C with a ViT-B-16 backbone.
  • The performance gains are more substantial with larger ViT backbones (ViT-S, ViT-B, ViT-L).
  • SoTa-DiT consistently outperforms other methods across most corruption types on ImageNet-C, excelling on 10 out of 15.
  • Ablation studies confirm the effectiveness of both the source and target prompts individually, with their joint use yielding the best results. Combining SP and TP after the softmax layer provides a significant boost over averaging before the classification head.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper investigates the influence of various hyperparameter weights (e.g., for loss functions) and notes that performance can be sensitive to their precise tuning. For instance, overly emphasizing source contrastive loss can harm adaptation, while too much target guiding loss can hinder effective extraction.
  • Future Work: The authors suggest that further exploration into optimizing the combination strategy of SP and TP predictions could yield even better results. Additionally, investigating more sophisticated prompt interaction mechanisms or exploring other architectures beyond ViT could be promising avenues.

Summary updated

BibTeX
@inproceedings{ma2025disentangle,
  author = {Tianyi Ma and Maoying Qiao},
  title = {{Disentangle Source and Target Knowledge for Continual Test-Time Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure