Self-supervised Test-time Adaptation on Video Data

Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jörn Hees, Luca Bertinetto, Andreas Dengel

Abstract

Standard computer vision models often fail when encountering real-world environmental shifts not present in their training data. This paper investigates Self-supervised Test-time Adaptation (TTA) as a solution for video object segmentation. By updating model statistics or weights during inference using only unlabeled video streams, the authors demonstrate significant recovery of tracking performance under severe corruptions like fog, snow, and motion blur.

Introduction: The Challenge of Covariate Shift

Most computer vision models are trained on clean, curated datasets. However, deployment often occurs in unpredictable environments—a model trained on sunny streets may struggle in a London fog or a Canadian blizzard. This phenomenon, known as covariate shift, typically requires expensive manual re-labeling and fine-tuning. For video data, where pixel-perfect masks are required for every frame, this cost is prohibitive. This research explores whether a model can adapt to a specific video stream at test time using only the video itself, without human intervention.

Methods for Rapid Adaptation

The authors propose a model-agnostic framework applying three adaptation strategies to dense tracking tasks:

  1. Prediction-time Batch Normalization (BN): Instead of static training statistics, the model calculates mean and variance from the current test video. A momentum-based update blends these statistics to account for the high correlation between consecutive video frames.
  2. TENT* (Entropy Minimization): This method adapts the "modulation parameters" (gain and shift) of BN layers by minimizing the model's own self-supervised tracking objective.
  3. Test-time Training (TTT): The entire feature extractor (backbone) is updated through a few iterations of a self-supervised task on the test video before final prediction.

These strategies were evaluated using state-of-the-art self-supervised trackers, VideoWalk and MAST.

Datasets and Experimental Setup

The study utilized DAVIS2017 for rapid adaptation on short clips and TAO-VOS for evaluating long-term stability over 1000+ frames. To simulate environmental shifts, five "level 5" severity filters were applied: Gaussian Noise, Motion Blur, Fog, and Snow.

Visualizing environmental corruptions: The original clean frame (top left) compared with snow (top right) and fog (bottom left) simulations.

Key Insights and Results

The research yielded several critical findings regarding the nature of video adaptation:

The Importance of Momentum

Unlike image-based TTA, which assumes independent and identically distributed (i.i.d.) samples, video frames are highly redundant. The authors found that a full replacement of training statistics (momentum = 1.0) often degrades performance. A partial, momentum-based update is necessary to maintain stability.

Impact of momentum on tracking performance (J&F score) across various noise types. Performance generally peaks at intermediate momentum values and drops sharply at 1.0.

Performance Recovery

Adaptation proved most beneficial in harsh environments. For instance, on the DAVIS2017 dataset under fog, the VideoWalk model’s J-score improved from 19.27 to 37.97 (+18.70 points) using TTT. Similarly, the MAST model saw improvements of over 20 points in the presence of heavy Gaussian noise.

The Proxy Gap

Interestingly, on clean data, adaptation can sometimes slightly decrease performance. This "proxy gap" occurs when the self-supervised objective used for adaptation does not perfectly align with the ground-truth tracking task, causing the model to over-optimize for a secondary goal.

Conclusion and Future Directions

Self-supervised TTA offers a powerful, label-free way to make video models robust to environmental changes. While methods like TTT and TENT* provide the best recovery for severe shifts, they currently introduce significant computational overhead. Future work remains to bridge the "proxy gap" and optimize these methods for real-time applications like autonomous driving and robotics, where temporal consistency can be further exploited for even more robust adaptation.

Summary updated

BibTeX
@inproceedings{azimi2022self,
  author = {Fatemeh Azimi and Sebastian Palacio and Federico Raue and Jörn Hees and Luca Bertinetto and Andreas Dengel},
  title = {{Self-supervised Test-time Adaptation on Video Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00266},
}
Figure