TIPI: Test Time Adaptation with Transformation Invariance

A. Tuan Nguyen, Thanh Nguyen-Tang, Ser-Nam Lim, Philip H. S. Torr

Abstract

Test-time adaptation (TTA) allows machine learning models to adjust to distribution shifts during deployment using unlabeled data. However, existing entropy-minimization methods like TENT often suffer from model collapse when processing small batches. We present TIPI (Test tIme adaPtation with transformation Invariance), a framework that enforces consistency across local perturbations. By utilizing adversarial noise as a proxy for distribution shifts and a reverse KL divergence loss, TIPI maintains high performance even with batch sizes as small as two, significantly outperforming prior state-of-the-art methods in real-world, low-latency scenarios.

Introduction

Machine learning models typically assume that deployment data follows the same distribution as training data. In practice, "distribution shift"—such as a self-driving car encountering snow for the first time—can severely degrade performance. Test-Time Adaptation (TTA) addresses this by updating the model on-the-fly using unlabeled data from the new environment.

The current state-of-the-art method, TENT, minimizes prediction entropy. While effective for large batches, TENT is prone to "model collapse" in small-batch settings typical of real-time applications like robotics. In these scenarios, the model may adopt a degenerate solution, such as predicting a single class for all inputs, to satisfy the entropy objective. TIPI is designed to overcome this limitation by prioritizing transformation invariance over simple certainty.

Methods

TIPI introduces an invariance regularizer that forces the model to produce consistent predictions even when inputs are slightly perturbed.

Transformation Invariance and Adversarial Proxies

Instead of relying on specific data augmentations (like rotations), TIPI uses $l_\infty$-norm adversarial noise generated via the Fast Gradient Sign Method (FGSM). This noise serves as a task-agnostic proxy for various distribution shifts. If a model is invariant to these "worst-case" local changes, it tends to be more robust to global shifts like weather or sensor noise.

The Reverse KL Trick and Selective Updates

The framework employs Reverse KL Divergence as its loss function. Unlike the standard Forward KL, Reverse KL "squashes" gradients, which reduces variance and prevents the model from over-correcting based on noisy, small-batch data. To maintain stability and computational efficiency, TIPI only updates the affine parameters (scale and shift) of the model's normalization layers.

Algorithm 1: The ADAPT-PREDICT procedure for TIPI.

As shown in the algorithm above, the original prediction $p_i$ is "detached" (gradient blocking) when calculating the invariance loss. This ensures the transformed prediction $p_\theta(y|x'_i)$ is forced to match the stable original prediction, preventing the model from trivializing the loss by making both predictions equally poor.

Experimental Results

TIPI was evaluated across several benchmarks, including CIFAR-C, ImageNet-C, and domain adaptation tasks like SVHN-to-MNIST.

Robustness to Small Batch Sizes

A key finding is TIPI's superior performance in low-data regimes. On CIFAR10-C with a batch size of 2, TENT's accuracy plummeted to 17.4%, indicating model collapse. In contrast, TIPI maintained a robust 78.6% accuracy.

Accuracy comparison on CIFAR10-C across different test-time adaptation batch sizes.

Large-Scale and Domain Adaptation

On ImageNet-C, TIPI achieved an average accuracy of 55.0%, outperforming TENT (45.2%) and the non-adapted source model (39.5%). In the SVHN-to-MNIST digit adaptation task, TIPI reached 86.3% accuracy with a batch size of 4, nearly tripling the performance of TENT (27.0%) in the same setting.

Discussion and Conclusion

TIPI demonstrates that consistency is a more reliable signal for adaptation than certainty, especially when data is sparse. While the adversarial perturbation step makes TIPI approximately twice as slow as TENT, it remains capable of real-time performance (160 FPS). Future research may explore combining TIPI with intelligent data selection strategies or domain-specific augmentations to further enhance robustness and efficiency in specialized applications like autonomous navigation.

Summary updated

Abstract

When deploying a machine learning model to a new environment, we often encounter the distribution shift problem - meaning the target data distribution is different from the model's training distribution. In this paper, we assume that labels are not provided for this new domain, and that we do not store the source data (e.g., for privacy reasons). It has been shown that even small shifts in the data distribution can affect the model's performance severely. Test Time Adaptation offers a means to combat this problem, as it allows the model to adapt during test time to the new data distribution, using only unlabeled test data batches. To achieve this, the predominant approach is to optimize a surrogate loss on the test-time unlabeled target data. In particular, minimizing the prediction's entropy on target samples [34] has received much interest as it is task-agnostic and does not require altering the model's training phase (e.g., does not require adding a self-supervised task during training on the source domain). However, as the target data's batch size is often small in real-world scenarios (e.g., autonomous driving models process each few frames in real-time), we argue that this surrogate loss is not optimal since it often collapses with small batch sizes. To tackle this problem, in this paper, we propose to use an invariance regularizer as the surrogate loss during test-time adaptation, motivated by our theoretical results regarding the model's performance under input transformations. The resulting method (TIPI-Test time adaPtation with transformation Invariance) is validated with extensive experiments in various benchmarks (Cifar10-C, Cifar100-C, ImageNet-C, DIGITS, and VisDA17). Remarkably, TIPI is robust against small batch sizes (as small as 2 in our experiments), and consistently outperforms TENT [34] in all settings. Our code is released at https://github.com/atuannguyen/TIPI.
BibTeX
@inproceedings{nguyen2023tipi,
  author = {A. Tuan Nguyen and Thanh Nguyen-Tang and Ser-Nam Lim and Philip H. S. Torr},
  title = {{TIPI: Test Time Adaptation with Transformation Invariance}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02314},
}
Figure