Abstract
Test-time adaptation (TTA) allows machine learning models to adjust to distribution shifts during deployment using unlabeled data. However, existing entropy-minimization methods like TENT often suffer from model collapse when processing small batches. We present TIPI (Test tIme adaPtation with transformation Invariance), a framework that enforces consistency across local perturbations. By utilizing adversarial noise as a proxy for distribution shifts and a reverse KL divergence loss, TIPI maintains high performance even with batch sizes as small as two, significantly outperforming prior state-of-the-art methods in real-world, low-latency scenarios.
Introduction
Machine learning models typically assume that deployment data follows the same distribution as training data. In practice, "distribution shift"—such as a self-driving car encountering snow for the first time—can severely degrade performance. Test-Time Adaptation (TTA) addresses this by updating the model on-the-fly using unlabeled data from the new environment.
The current state-of-the-art method, TENT, minimizes prediction entropy. While effective for large batches, TENT is prone to "model collapse" in small-batch settings typical of real-time applications like robotics. In these scenarios, the model may adopt a degenerate solution, such as predicting a single class for all inputs, to satisfy the entropy objective. TIPI is designed to overcome this limitation by prioritizing transformation invariance over simple certainty.
Methods
TIPI introduces an invariance regularizer that forces the model to produce consistent predictions even when inputs are slightly perturbed.
Transformation Invariance and Adversarial Proxies
Instead of relying on specific data augmentations (like rotations), TIPI uses $l_\infty$-norm adversarial noise generated via the Fast Gradient Sign Method (FGSM). This noise serves as a task-agnostic proxy for various distribution shifts. If a model is invariant to these "worst-case" local changes, it tends to be more robust to global shifts like weather or sensor noise.
The Reverse KL Trick and Selective Updates
The framework employs Reverse KL Divergence as its loss function. Unlike the standard Forward KL, Reverse KL "squashes" gradients, which reduces variance and prevents the model from over-correcting based on noisy, small-batch data. To maintain stability and computational efficiency, TIPI only updates the affine parameters (scale and shift) of the model's normalization layers.
As shown in the algorithm above, the original prediction $p_i$ is "detached" (gradient blocking) when calculating the invariance loss. This ensures the transformed prediction $p_\theta(y|x'_i)$ is forced to match the stable original prediction, preventing the model from trivializing the loss by making both predictions equally poor.
Experimental Results
TIPI was evaluated across several benchmarks, including CIFAR-C, ImageNet-C, and domain adaptation tasks like SVHN-to-MNIST.
Robustness to Small Batch Sizes
A key finding is TIPI's superior performance in low-data regimes. On CIFAR10-C with a batch size of 2, TENT's accuracy plummeted to 17.4%, indicating model collapse. In contrast, TIPI maintained a robust 78.6% accuracy.
Large-Scale and Domain Adaptation
On ImageNet-C, TIPI achieved an average accuracy of 55.0%, outperforming TENT (45.2%) and the non-adapted source model (39.5%). In the SVHN-to-MNIST digit adaptation task, TIPI reached 86.3% accuracy with a batch size of 4, nearly tripling the performance of TENT (27.0%) in the same setting.
Discussion and Conclusion
TIPI demonstrates that consistency is a more reliable signal for adaptation than certainty, especially when data is sparse. While the adversarial perturbation step makes TIPI approximately twice as slow as TENT, it remains capable of real-time performance (160 FPS). Future research may explore combining TIPI with intelligent data selection strategies or domain-specific augmentations to further enhance robustness and efficiency in specialized applications like autonomous navigation.
Summary updated