Bag of Tricks for Fully Test-Time Adaptation

Saypraseuth Mounsaveng, Florent Chiaroni, Malik Boudiaf, Marco Pedersoli, Ismail Ben Ayed

Abstract

Fully Test-Time Adaptation (TTA) is the process of updating a pre-trained model using only a stream of unlabeled, shifted test data. While promising, existing TTA methods are often brittle when faced with realistic "in-the-wild" conditions such as small batch sizes and severe class imbalance. This paper provides a systematic dissection of TTA and identifies a "Bag of Tricks" (BoT)—including batch renormalization, class re-balancing, and confidence calibration—that establishes new state-of-the-art results and makes TTA significantly more robust for real-world deployment.

Introduction

Deep learning models frequently encounter distribution shifts in deployment, where test data differs from training data (e.g., weather effects or stylistic changes). Fully TTA aims to mitigate this on-the-fly without requiring source data or labels. However, standard TTA methods like Tent assume ideal conditions, such as large, perfectly balanced batches. In real-world scenarios, data often arrives in tiny batches or even single instances, leading to model collapse where the model performs worse after adaptation than before.

Methods

The authors evaluate and combine several orthogonal techniques to form the "Bag of Tricks" (BoT):

  1. Normalization Tweaks: Standard Batch Normalization (BN) fails on small batches due to high variance in statistics. The authors suggest using Batch Renormalization (BReN) or architectures using Group Normalization (GN) or Layer Normalization (LN).
  2. Class Re-balancing (DOT): Diversity-Oriented Test-time adaptation (DOT) estimates class frequencies on-the-fly and weights updates to prevent bias toward frequent classes in the stream.
  3. Reliable Sample Selection: An entropy-based filter selects only high-confidence samples for adaptation, improving stability and reducing computational overhead.
  4. Temperature Scaling: A simple calibration trick that adjusts output confidence to prevent the model from becoming overconfident in its own potentially incorrect predictions.
Figure 3: Accuracy vs. Imbalance Factor across different architectures and batch sizes. DOT (blue) consistently improves robustness to class imbalance compared to Tent (red), especially as the imbalance factor increases.

Key Insights

  • The BatchNorm Trap: BN is the primary failure point for TTA at small batch sizes. When the batch size is 1 or 2, the statistics are so noisy they "poison" the model's weights. Switching to BReN or LN is a critical requirement for robust TTA.
  • Architecture Robustness: Vision Transformers (ViTs) using Layer Normalization are inherently more robust to small batch sizes and class imbalance than traditional ResNets, even before applying TTA tricks.
  • Synergy and Efficiency: The tricks are more than the sum of their parts. By using sample selection, the authors achieved superior results while only performing backpropagation on approximately 26% of the data, significantly lowering the power requirements for on-device adaptation.

Key Results

The BoT consistently outperformed previous state-of-the-art methods like Tent, SAR, and Delta across major benchmarks including ImageNet-C, ImageNet-R, and VisDA2017.

Figure 1: Accuracy vs. Batch Size. The Bag of Tricks (BoT, starred lines) maintains high performance across all batch sizes, preventing the collapse seen in methods like Tent and SAR as the batch size decreases to 1.

On ImageNet-C with a batch size of 1, a standard ResNet50 using Tent collapses to 0.14% accuracy. Applying the BoT brings that up to 20.31%. Across all batch sizes and architectures, BoT provided an average accuracy improvement of +17.08% over vanilla Tent on ResNet50-BN and +7.66% on Vision Transformers.

Conclusion

By identifying and combining simple yet effective tricks, this work makes Fully TTA practical for real-world applications where data streams are unpredictable and resource-constrained. The resulting "Bag of Tricks" establishes a new baseline for robust, efficient adaptation in the wild.

Summary updated

BibTeX
@inproceedings{mounsaveng2024bag,
  author = {Saypraseuth Mounsaveng and Florent Chiaroni and Malik Boudiaf and Marco Pedersoli and Ismail Ben Ayed},
  title = {{Bag of Tricks for Fully Test-Time Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=fa0Oaa2yCL},
}
Figure