PROGRAM: PROtotype GRAph Model based Pseudo-Label Learning for Test-Time Adaptation

Haopeng Sun, Lumin Xu, Sheng Jin, Ping Luo, Chen Qian, Wentao Liu

TL;DR

PROGRAM is a novel test-time adaptation (TTA) framework designed to improve model performance on new, unlabeled data. It generates highly reliable pseudo-labels through a graph-based propagation model that combines global class prototypes with local sample relationships. Coupled with a noise-robust self-training strategy, PROGRAM achieves state-of-the-art results across major domain generalization and image corruption benchmarks.

The Challenge: Domain Shift and Noisy Pseudo-Labels

Deep learning models are notoriously brittle when the data they encounter at "test-time" differs from their training data—a phenomenon known as domain shift. Test-Time Adaptation (TTA) aims to fix this by updating the model on-the-fly using only the unlabeled stream of incoming data.

A popular strategy for TTA is pseudo-labeling: the model predicts a label for an unlabeled image and then treats that prediction as "ground truth" to fine-tune itself. The catch? If the model is wrong (which is highly likely under domain shift), it reinforces its own errors. This "garbage in, garbage out" cycle can cause the model to collapse entirely. Existing fixes often rely on hand-picked confidence thresholds that are difficult to tune for every possible scenario.

The PROGRAM Approach

To tackle the noise problem, the authors introduce PROGRAM (PROtotype GRAph Model), which consists of two main components:

PROGRAM Architecture
  1. Prototype Graph Model (PGM): Instead of relying on a single image's prediction, PGM builds a graph for every batch of data. The graph contains two types of nodes: Class Prototypes (global representations of classes, derived from the classifier's weights) and Test Samples (the actual images in the current batch). By performing label propagation on this graph, the model "spreads" label information from the reliable prototypes to the test samples based on their feature similarity. This allows the model to consider both global class structure and local data clusters simultaneously.
  2. Robust Self-Training (RST): Even with a graph, some labels might remain noisy. RST decides how to learn from a sample based on consistency. If the original classifier and the PGM graph agree on a label, the model uses a standard "hard" pseudo-label to accelerate learning. If they disagree, the model treats the label as potentially noisy and applies Symmetric Cross Entropy (SCE) loss. SCE is a noise-robust loss function that prevents the model from over-fitting to incorrect labels while still extracting useful information from the data.

Key Insights and Advantages

Comparison of Pseudo-Labeling Strategies
  • Global + Local > Either Alone: As illustrated above, previous methods either used global prototypes (missing local data nuances) or nearest neighbors (getting confused by outliers). PROGRAM’s graph structure effectively bridges this gap, leveraging both global prototypes and local neighborhood structures for more accurate classification boundaries.
  • Agreement as a Proxy for Confidence: Instead of setting a manual threshold (e.g., "only learn if confidence > 0.9"), PROGRAM uses the consensus between the classifier and the graph. This makes the system much more flexible across different datasets without requiring manual tuning.
  • Plug-and-Play Improvement: One of the most interesting findings is that the PGM module can be "plugged into" other existing TTA methods (like T3A or TAST). In every tested case, replacing their default label generation with PGM led to a performance boost.

Experimental Setup and Results

The authors evaluated PROGRAM on two main types of challenges:

  • Domain Generalization Benchmarks: VLCS, PACS, OfficeHome, and TerraIncognita. These test the model's ability to move from one style (e.g., photos) to another (e.g., sketches or cartoons).
  • Image Corruption Benchmarks: CIFAR-10C, CIFAR-100C, and ImageNet-C, which involve 15 types of digital corruptions (blur, noise, weather effects) at various severity levels.

Key Results: * State-of-the-Art Performance: On the PACS benchmark (using ResNet-50), PROGRAM achieved 90.78% accuracy, significantly outperforming the baseline ERM (83.21%) and other SOTA methods like TSD (89.27%). * Robustness to Corruption: On image corruption benchmarks, PROGRAM reduced the average error rate by 23.91% compared to the non-adapted baseline. * Efficiency: Despite the graph construction, PROGRAM remains computationally efficient. Its runtime is comparable to other "whole-model" update methods but delivers much higher accuracy. * Feature Clustering: t-SNE visualizations demonstrated that PROGRAM produces much tighter, more distinct clusters in the feature space compared to previous methods, proving it learns more discriminative representations under shift.

Limitations and Future Work

The authors acknowledge a few key areas for future growth:

  • Task Scope: Currently, PROGRAM is designed for classification. Extending this graph-based approach to regression tasks, object detection, or semantic segmentation remains an open challenge.
  • Evaluation in the Wild: Like all TTA methods that update model parameters continuously, it is difficult to guarantee the model won't "drift" or behave unpredictably over very long periods of deployment.
  • Ethical Considerations: Constant model updates in sensitive applications (like medical imaging) could raise concerns about consistency and reliability that require further study before real-world deployment.

Summary updated

Abstract

Test-time adaptation (TTA) aims to adapt a pre-trained model from a source domain to a target domain only using online unlabeled target data during testing, without accessing to the source data or modifying the original training process. Among the various TTA methods, pseudo-labeling has gained popularity. However, the presence of incorrect pseudo-labels can hinder the effectiveness of target domain adaptation. To overcome this challenge, we propose a novel TTA method, called PROtotype GRAph Model based pseudo-label learning (PROGRAM). PROGRAM consists of two key components: (1) Prototype Graph Model (PGM) for reliable pseudo-label generation; (2) Robust Self-Training (RST) for test-time adaptation with noisy pseudo-labels. PGM constructs the graph using prototypes and test samples, facilitating effective message passing among them to generate more reliable pseudo-labels. RST combines the advantages of consistency regularization and pseudo-labeling to achieve robust target domain adaptation in the presence of noisy pseudo-labels. Our proposed PROGRAM can be easily integrated into existing baselines, resulting in consistent improvement. Extensive experiments show that our PROGRAM outperforms the existing TTA methods on multiple domain generalization and image corruption benchmarks.
BibTeX
@inproceedings{sun2024program,
  author = {Haopeng Sun and Lumin Xu and Sheng Jin and Ping Luo and Chen Qian and Wentao Liu},
  title = {{PROGRAM: PROtotype GRAph Model based Pseudo-Label Learning for Test-Time Adaptation}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=x5LvBK43wg},
}
Figure