NC-TTT: A Noise Constrastive Approach for Test-Time Training

David Osowiechi, Gustavo Adolfo Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, Christian Desrosiers

Abstract

Deep learning models often suffer from performance degradation when encountering domain shifts at test time. Test-Time Training (TTT) addresses this by updating model parameters using an auxiliary task on unlabeled test data. We introduce Noise-Contrastive Test-Time Training (NC-TTT), which employs a discriminator to differentiate between source-like features and noisy versions. This approach allows the model to align test-time features with the training distribution in a principled manner. NC-TTT demonstrates significant accuracy gains on CIFAR-10-C, CIFAR-100-C, and VisDA-C, outperforming existing TTT methods.

Introduction

Deep learning models are notoriously fragile when encountering data that differs from their training distribution. A model trained on clear, sunny photos might fail completely when deployed in grainy, low-light environments or when processing stylized renderings—a phenomenon known as domain shift. While predicting every possible real-world distortion is impossible, Test-Time Training (TTT) offers a solution by allowing models to adapt to new, unlabeled data during deployment through an auxiliary "side quest."

Traditional TTT auxiliary tasks, such as predicting image rotation, can be arbitrary. NC-TTT seeks a more principled approach by using Noise-Contrastive Estimation (NCE) to align test-time features with the original training distribution, effectively re-tuning the model's encoder to the new domain without requiring labels.

Methodology

NC-TTT utilizes a "Y-shaped" network architecture where a primary branch handles the main task (e.g., classification) and an auxiliary branch manages the adaptation task.

Overview of the NC-TTT architecture, featuring a main classification branch and an auxiliary noise-contrastive branch.

The auxiliary task involves projecting internal feature maps into a lower-dimensional space and creating two versions: an "in-distribution" version with minimal noise ($\sigma_s$) and an "out-of-distribution" (OOD) version with significant noise ($\sigma_o$). A lightweight discriminator is trained to distinguish between these views. During test-time adaptation, the model encounters shifted data and updates its encoder parameters so that the resulting features appear "in-distribution" to the pre-trained discriminator. This process essentially treats the discriminator as a proxy for the probability density of the source data, pushing test features into the "safe zone" of the original distribution.

Experiments and Results

The researchers evaluated NC-TTT on three major domain-shift benchmarks: CIFAR-10-C, CIFAR-100-C, and the VisDA-C sim-to-real dataset.

Accuracy improvement across various image corruptions over adaptation iterations.

NC-TTT demonstrated state-of-the-art performance across these benchmarks:

  • CIFAR-10-C: Achieved 82.43% average accuracy, a significant improvement over the non-adapted ResNet50 baseline of 53.82%.
  • CIFAR-100-C: Reached 57.68% accuracy, compared to the 31.37% baseline.
  • VisDA-C: On this difficult synthetic-to-real task, NC-TTT achieved 62.71% accuracy, surpassing other popular methods like TENT (60.34%) and standard TTT (40.57%).

As shown in the results, the model consistently recovers accuracy across a wide variety of corruptions—including Gaussian noise, motion blur, and snow—within a few dozen iterations of adaptation.

Discussion and Conclusion

NC-TTT provides a principled framework for test-time adaptation by framing the auxiliary task as a density estimation problem. The use of a small amount of in-distribution noise prevents over-adaptation, ensuring the model remains grounded in the source distribution. While highly effective and computationally efficient due to its lightweight auxiliary module, future work remains to explore different noise types and the scaling of this approach to larger architectures like Transformers. NC-TTT represents a significant step toward making deep learning models robust enough for unpredictable real-world deployment.

Summary updated

Abstract

Despite their exceptional performance in vision tasks, deep learning models often struggle when faced with domain shifts during testing. Test-Time Training (TTT) methods have recently gained popularity by their ability to enhance the robustness of models through the addition of an auxiliary objective that is jointly optimized with the main task. Being strictly unsupervised, this auxiliary objective is used at test time to adapt the model without any access to labels. In this work, we propose Noise-Contrastive TestTime Training (NC-TTT), a novel unsupervised TTT technique based on the discrimination of noisy feature maps. By learning to classify noisy views of projected feature maps, and then adapting the model accordingly on new domains, classification performance can be recovered by an important margin. Experiments on several popular testtime adaptation baselines demonstrate the advantages of our method compared to recent approaches for this task. The code can be found at: https://github.com/GustavoVargasHakim/NCTTT.git
BibTeX
@inproceedings{osowiechi2024nc,
  author = {David Osowiechi and Gustavo Adolfo Vargas Hakim and Mehrdad Noori and Milad Cheraghalikhani and Ali Bahri and Moslem Yazdanpanah and Ismail Ben Ayed and Christian Desrosiers},
  title = {{NC-TTT: A Noise Constrastive Approach for Test-Time Training}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00581},
}
Figure