Abstract
Fully Test-Time Adaptation (FTTA) enables machine learning models to adapt to target domains on-the-fly without access to original source training data. However, standard entropy minimization techniques often lead to a "diversity gap," where models become overconfident in easy classes while ignoring harder ones, and suffer from overfitting due to small batch sizes. This paper introduces a dual-strategy approach: Singular Value Penalization (SVP) to equalize class attention and Semantic Data Augmentation (SDA) to stabilize adaptation using historical feature statistics. Our method achieves state-of-the-art robustness on major out-of-distribution benchmarks including CIFAR-C and ImageNet-C.
Introduction
In real-world deployments, models frequently encounter domain shifts—data distributions that differ from the training set. Fully Test-Time Adaptation (FTTA) addresses this by adapting the model during the testing phase. While effective, current methods primarily rely on entropy minimization, which encourages the model to be more certain about its predictions. This often results in two critical failures:
- The Diversity Gap: The model prioritizes "easy" classes with strong signals, effectively drowning out minority or "hard" classes.
- Overfitting: Adapting to small, noisy batches can cause the model to lose its generalizability.
Methodology
Singular Value Penalization (SVP)
To address the diversity gap, we apply Singular Value Decomposition (SVD) to the model's prediction matrix. Our optimization objective is twofold:
- Maximizing the Sum of Singular Values: This improves both the overall confidence and the diversity of predicted labels.
- Minimizing the Variance of Singular Values: This is our core innovation. In standard adaptation, a few large singular values (representing easy classes) dominate. By forcing the variance of these values to be small, we compel the model to pay attention to smaller singular values, which represent more challenging, less distinguishable classes.
Semantic Data Augmentation (SDA)
To mitigate overfitting to individual batches, we incorporate a "memory" mechanism. We calculate the running mean and covariance of features for each class across previous batches. These statistics are used to semantically perturb the current batch's features—effectively changing the "style" of a sample while preserving its class identity. This provides the model with a richer set of training samples without requiring additional real-world data.
Experimental Results
We evaluated our approach on gold-standard benchmarks for model robustness, including CIFAR-10-C, CIFAR-100-C, and ImageNet-C, using architectures such as WideResNet and ResNet-50.
Quantitative Performance
Our method consistently outperformed established baselines like TENT and DUA:
- CIFAR-10-C: Achieved a Top-1 error of 16.5% (WRN-28-10), compared to TENT's 18.6%.
- ImageNet-C: On a ResNet-50 backbone, we reached 55.8% error, a significant improvement over the Source model's 92.7% and TENT's 57.4%.
Qualitative Analysis
t-SNE visualizations demonstrate that our method produces much cleaner and more distinct class clusters in the target domain compared to existing adaptation techniques.
Discussion and Insights
The success of our method highlights that singular values act as class proxies. Smaller singular values often represent difficult-to-distinguish class combinations; by penalizing variance, we prevent the "rich get richer" problem in AI where strong signals dominate the learning process. Furthermore, the use of online estimation for class statistics stabilizes the adaptation process, making it robust against the noise inherent in small-batch testing.
Conclusion
By combining Singular Value Penalization with Semantic Data Augmentation, we provide a robust framework for Fully Test-Time Adaptation that maintains class diversity and prevents overfitting. Future work will explore integrating this approach as a plug-and-play module for other optimization-based adaptation tasks.
Summary updated