Singular Value Penalization and Semantic Data Augmentation for Fully Test-Time Adaptation

Houcheng Su, Daixian Liu, Mengzhu Wang, Wei WangarXiv 2023 · DOI · arXiv

Abstract

Fully Test-Time Adaptation (FTTA) enables machine learning models to adapt to target domains on-the-fly without access to original source training data. However, standard entropy minimization techniques often lead to a "diversity gap," where models become overconfident in easy classes while ignoring harder ones, and suffer from overfitting due to small batch sizes. This paper introduces a dual-strategy approach: Singular Value Penalization (SVP) to equalize class attention and Semantic Data Augmentation (SDA) to stabilize adaptation using historical feature statistics. Our method achieves state-of-the-art robustness on major out-of-distribution benchmarks including CIFAR-C and ImageNet-C.

Introduction

In real-world deployments, models frequently encounter domain shifts—data distributions that differ from the training set. Fully Test-Time Adaptation (FTTA) addresses this by adapting the model during the testing phase. While effective, current methods primarily rely on entropy minimization, which encourages the model to be more certain about its predictions. This often results in two critical failures:

  1. The Diversity Gap: The model prioritizes "easy" classes with strong signals, effectively drowning out minority or "hard" classes.
  2. Overfitting: Adapting to small, noisy batches can cause the model to lose its generalizability.

Methodology

Singular Value Penalization (SVP)

To address the diversity gap, we apply Singular Value Decomposition (SVD) to the model's prediction matrix. Our optimization objective is twofold:

  • Maximizing the Sum of Singular Values: This improves both the overall confidence and the diversity of predicted labels.
  • Minimizing the Variance of Singular Values: This is our core innovation. In standard adaptation, a few large singular values (representing easy classes) dominate. By forcing the variance of these values to be small, we compel the model to pay attention to smaller singular values, which represent more challenging, less distinguishable classes.
Conceptual illustration of how Singular Value Penalization balances class diversity by reducing the variance between singular values.

Semantic Data Augmentation (SDA)

To mitigate overfitting to individual batches, we incorporate a "memory" mechanism. We calculate the running mean and covariance of features for each class across previous batches. These statistics are used to semantically perturb the current batch's features—effectively changing the "style" of a sample while preserving its class identity. This provides the model with a richer set of training samples without requiring additional real-world data.

Experimental Results

We evaluated our approach on gold-standard benchmarks for model robustness, including CIFAR-10-C, CIFAR-100-C, and ImageNet-C, using architectures such as WideResNet and ResNet-50.

Quantitative Performance

Our method consistently outperformed established baselines like TENT and DUA:

  • CIFAR-10-C: Achieved a Top-1 error of 16.5% (WRN-28-10), compared to TENT's 18.6%.
  • ImageNet-C: On a ResNet-50 backbone, we reached 55.8% error, a significant improvement over the Source model's 92.7% and TENT's 57.4%.

Qualitative Analysis

t-SNE visualizations demonstrate that our method produces much cleaner and more distinct class clusters in the target domain compared to existing adaptation techniques.

t-SNE visualizations comparing the proposed method against baselines, demonstrating superior feature clustering in the target domain.

Discussion and Insights

The success of our method highlights that singular values act as class proxies. Smaller singular values often represent difficult-to-distinguish class combinations; by penalizing variance, we prevent the "rich get richer" problem in AI where strong signals dominate the learning process. Furthermore, the use of online estimation for class statistics stabilizes the adaptation process, making it robust against the noise inherent in small-batch testing.

Conclusion

By combining Singular Value Penalization with Semantic Data Augmentation, we provide a robust framework for Fully Test-Time Adaptation that maintains class diversity and prevents overfitting. Future work will explore integrating this approach as a plug-and-play module for other optimization-based adaptation tasks.

Summary updated

Abstract

Fully test-time adaptation (FTTA) adapts a model that is trained on a source domain to a target domain during the testing phase, where the two domains follow different distributions and source data is unavailable during the training phase. Existing methods usually adopt entropy minimization to reduce the uncertainty of target prediction results, and improve the FTTA performance accordingly. However, they fail to ensure the diversity in target prediction results. Recent domain adaptation study has shown that maximizing the sum of singular values of prediction results can simultaneously enhance their confidence (discriminability) and diversity. However, during the training phase, larger singular values usually take up a dominant position in loss maximization. This results in the model being more inclined to enhance discriminability for easily distinguishable classes, and the improvement in diversity is insufficiently effective. Furthermore, the adaptation and prediction in FTTA only use data from the current batch, which may lead to the risk of overfitting. To address the aforementioned issues, we propose maximizing the sum of singular values while minimizing their variance. This enables the model's focus toward the smaller singular values, enhancing discriminability between more challenging classes and effectively increasing the diversity of prediction results. Moreover, we incorporate data from the previous batch to realize semantic data augmentation for the current batch, reducing the risk of overfitting. Extensive experiments on benchmark datasets show our proposed approach outperforms some compared state-of-the-art FTTA methods.
BibTeX
@misc{su2023singular,
  author = {Houcheng Su and Daixian Liu and Mengzhu Wang and Wei Wang},
  title = {{Singular Value Penalization and Semantic Data Augmentation for Fully Test-Time Adaptation}},
  year = {2023},
  eprint = {2312.08378},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.08378},
}
Figure