Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection

Wenxuan Bao, Yanjun Zhao, Xiyuan Yang, Jingrui He

TL;DR

Ramen introduces a robust framework for test-time adaptation (TTA) of vision-language models facing data streams from mixed, unpredictable domains. By actively selecting a customized, domain-consistent, and class-balanced support set for each test sample and utilizing an efficient embedding-gradient cache, Ramen achieves state-of-the-art accuracy under complex distribution shifts without the massive computational overhead typical of per-sample adaptation.

What Problem Does This Paper Address?

Pretrained Vision-Language Models (VLMs) like CLIP excel at zero-shot generalization but remain highly sensitive to distribution shifts, such as changes in lighting, weather, or image corruptions. Test-Time Adaptation (TTA) has emerged as a popular solution to update models on-the-fly during deployment using unlabeled test data.

However, almost all existing TTA methods operate under a restrictive assumption: that the incoming test data arrives from a single, consistent domain at any given time. In real-world deployments—such as a smartphone photo gallery app—the data stream is often a chaotic mixture of domains interleaved together. When conventional TTA algorithms encounter these mixed-domain streams, their performance degrades significantly. This happens because a single model is forced to adapt to an "averaged" representation of all domains simultaneously, effectively diluting the domain-specific signals needed for accurate specialization.

Methods

To prevent mixed-domain signals from muddying the adaptation process, the authors propose Ramen (Robust test-time adaptation with active sample selection). Instead of passively adapting to the entire incoming stream, Ramen actively constructs a sample-specific support set ($\sS_i$) tailored to each incoming test sample ($x_i$).

The support set selection relies on two core criteria:

  1. Domain Consistency: It retrieves past samples that are closest to the current sample in the VLM's image embedding space. Because VLM embeddings implicitly cluster by style and domain, this acts as an effective proxy for domain matching.
  2. Prediction Balance: To prevent the model from developing a bias toward dominant classes, Ramen maintains first-in-first-out (FIFO) queues for each class and extracts an equal number of samples from each queue to form the final support set.
Figure 1. Overview of Ramen. For each test sample, (1) compute its image embedding, pseudo-label, and sample-level gradient; (2) update the class-specific memory with these entries; (3) retrieve a support set from the memory; (4) aggregate the cached gradients for model update; and (5) perform inference and reset the model parameters.

The Embedding-Gradient Cache

Computing a full forward and backward pass over a retrieved support set for every single test sample is computationally prohibitive. To solve this, Ramen introduces an embedding-gradient cache. The framework caches both the image embeddings and the sample-level gradients of past test images. When a new sample arrives, Ramen simply retrieves the relevant past gradients from the cache and aggregates them (weighted by entropy and similarity) to perform a lightweight update on the model's normalization layers. This completely eliminates the need for extra forward or backward passes on the support set.

Datasets

The authors evaluated Ramen under fully interleaved mixed-domain settings across several standard benchmarks:

  • CIFAR-10-C (using a ViT-B/32 backbone)
  • CIFAR-100-C (using a ViT-B/16 backbone)
  • ImageNet-C (using a ViT-L/14 backbone)
  • DomainNet (using a ViT-B/32 backbone)

Unlike standard evaluations that test corruptions or domains sequentially, these experiments mixed all 15 corruptions (or 6 domains in DomainNet) randomly within the test stream.

Key Insights and Ideas

  • Theoretical Justification: The authors provide a mathematical analysis proving that entropy minimization alters feature importance based on target domain variance. In mixed-domain streams, domain-specific features exhibit artificially high variance, causing standard TTA to mistakenly amplify them.
  • Variance Suppression via Selection: By enforcing domain consistency, Ramen ensures that domain-relevant features remain consistent within the support set. This minimizes their variance, allowing the optimization step to effectively suppress them while amplifying class-discriminative features.
  • Embedding Space as a Domain Proxy: Nearest-neighbor retrieval in CLIP's embedding space is remarkably accurate at grouping domains without explicit labels. Empirically, 40.9% of Ramen's retrieved support samples come from the exact same domain as the query sample, compared to just 6.7% for random selection.
Figure 3. Visualization of active sample selection on CIFAR-100- C. Each entry (i, j) indicates the average proportion (%) of sup- port samples from domain j when the test sample comes from do- main i. On average, 40.9% of the support samples come from the same domain as the test sample, which is substantially higher than random selection (6.7%). Results on other datasets are provided in Appendix C.4.

Key Results

  • State-of-the-Art Accuracy: Ramen consistently outperforms all baseline TTA methods (including Tent, NOTE, SAR, RoTTA, and Mint) under mixed-domain shifts. It improves average accuracy by +1.3% on CIFAR-10-C, +3.4% on CIFAR-100-C, +2.5% on ImageNet-C, and +0.6% on DomainNet over the strongest baselines.
  • Robustness to Mixture: While competitive VLM-focused TTA methods experience severe performance drops when moving from single-domain to mixed-domain streams, Ramen maintains stable and high accuracy across both settings.
Figure 2. Performance comparison of TTA methods under single- domain and mixed-domain shifts on CIFAR-100-C. Results on other datasets are provided in Appendix C.3.
  • Massive Speedups: The embedding-gradient cache achieves a staggering 490× speedup compared to a naive implementation without caching, reducing the total testing time on CIFAR-100-C from over 115 hours down to just 14 minutes.

Limitations and Future Work

The primary trade-off in Ramen is memory overhead. Storing both embeddings and high-dimensional gradients in the GPU cache requires roughly 14.5 GB of GPU memory for the CIFAR-100-C setup.

To mitigate this, the authors demonstrated an alternative configuration that offloads gradients to the CPU, which slashes the GPU cache memory requirement to a negligible 73 MiB. However, this introduces CPU-GPU communication bottlenecks, increasing the runtime from 14 minutes to 46 minutes. Future work could explore more compact gradient representations or sketching techniques to optimize this memory-time trade-off further.

Summary updated

Abstract

Pretrained vision-language models such as CLIP exhibit strong zero-shot generalization but remain sensitive to distribution shifts. Test-time adaptation adapts models during inference without access to source data or target labels, offering a practical way to handle such shifts. However, existing methods typically assume that test samples come from a single, consistent domain, while in practice, test data often include samples from mixed domains with distinct characteristics. Consequently, their performance degrades under mixed-domain settings. To address this, we present Ramen, a framework for robust test-time adaptation through active sample selection. For each incoming test sample, Ramen retrieves a customized batch of relevant samples from previously seen data based on two criteria: domain consistency, which ensures that adaptation focuses on data from similar domains, and prediction balance, which mitigates adaptation bias caused by skewed predictions. To improve efficiency, Ramen employs an embedding-gradient cache that stores the embeddings and sample-level gradients of past test images. The stored embeddings are used to retrieve relevant samples, and the corresponding gradients are aggregated for model updates, eliminating the need for any additional forward or backward passes. Our theoretical analysis provides insight into why the proposed adaptation mechanism is effective under mixed-domain shifts. Experiments on multiple image corruption and domain-shift benchmarks demonstrate that Ramen achieves strong and consistent performance, offering robust and efficient adaptation in complex mixed-domain scenarios. Our code is available at https://github.com/baowenxuan/Ramen .
BibTeX
@inproceedings{bao2026ramen,
  author = {Wenxuan Bao and Yanjun Zhao and Xiyuan Yang and Jingrui He},
  title = {{Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=tWwPlOVpxv},
}
Figure