Retrieval-augmented test-time adaptation helps a vision-language model such as CLIP (a model that scores how well an image matches a text) recognise images from unfamiliar domains. At test time it looks up reference images of each candidate class in a large web database and compares the test image against them. The method works when the database contains good references. When it does not, it fails without warning. A satellite tile of a rare land-use class, or a photo of an uncommon aircraft variant, retrieves the nearest images the database happens to hold, and those distractors pull the prediction away from the right answer. This proposal targets that failure directly. We detect, per test query and per candidate class, when the database has no good coverage. Only for those holes, we generate replacement reference images with a text-to-image diffusion model, a generator that turns random noise into an image guided by a prompt. Two constraints make the generated images usable. First, each generated image is anchored to the visual style of the unlabeled target domain, so it does not look like a generic stock photo. Second, each generated image must pass a round-trip consistency check, which removes the generator's hallucinations before they can act as references.
The claim is narrow and testable. Under extreme domain shift, gated, target-styled and verified synthetic references recover most of the accuracy that retrieval loses when references are removed from the database. Unverified or unstyled synthetic references do not recover it. We test this with a controlled coverage-deletion protocol that, as far as we can find, no prior retrieval-based adaptation paper runs. The protocol turns the vague phrase "knowledge base coverage bottleneck" into a curve one can measure.
Where this sits in prior work
The base system is RA-TTA. It uses descriptions written offline by a large language model, such as "boomerang-shaped headlights", to retrieve LAION2B images. It then compares the test image with the retrieved images using optimal transport, a way of matching two sets of features at minimum total cost. It reports 73.09% average accuracy on 13 transfer datasets, against 66.05% for plain CLIP. Its own stated limits motivate this proposal. Performance is bounded by what LAION contains, and the descriptions can themselves be hallucinated. MOTOR provides parallel evidence from medicine. Cosine retrieval over MIMIC-CXR surfaces reports with unrelated pathologies, and optimal-transport re-ranking helps, but nothing can be re-ranked into existence when the index lacks the case. ImageRAG also stalls when a concept is missing from its 350K reference set. All three papers motivate the problem, and all three are purely extractive.
The closest generative bridge is ROSCap. It retrieves from a memory bank of Stable Diffusion images to close the gap between text and images in zero-shot captioning. Its bank, however, is synthesised in advance, it assumes the relevant concepts are already present, and it is evaluated only on natural images. We differ in three ways. We synthesise on demand at detected holes. We condition generation on target-domain style. We filter what we generate. RA-TTA's own comparison table already includes a pure synthetic-support baseline, SuS-X-LC, at 70.89%. That method builds class references from text-only generation, and it scores below retrieval. We read that number as a constraint. Synthetic references are not automatically better than real ones, so any gain we report must come from gating, style anchoring and verification, not from synthesis as such.
The verification step reuses technique from two papers. Triangular consistency keeps a synthetic image–question–answer triplet only if the answer and the question can each be regenerated from the other parts. VeriSciQA adds "visual dependence" blind tests and accepts that only 3.0% of candidates survive. We adapt both to synthetic reference images rather than question-answer data.
Two further papers constrain the design. The style-anchoring idea borrows from gradual domain adaptation. GOAT builds intermediate "stepping-stone" domains along an optimal-transport path, and STDW shows that errors in those intermediate steps compound. We take from them the idea of an interpolation path, and the warning that unverified intermediate samples poison later stages. SDA argues that diffusion outputs live on their own "synthetic domain", so comparing real queries against synthetic references is itself shifted. We test that concern directly as an ablation rather than assume it away.
Core mechanism
The pipeline has three stages, and each targets one failure we can name.
The first stage detects coverage holes. For a query image $x$ and candidate class $y$, RA-TTA already retrieves a reference set $R_y$ and computes an optimal-transport cost between the patch features of $x$ and those of $R_y$. We turn that cost into a coverage score $$c(x,y) = \frac{\mathrm{OT}(x, R_y)}{\mathrm{med}_{x' \in \mathcal{C}_y} \mathrm{OT}(x', R_y)},$$ where $\mathrm{OT}$ is the entropic transport cost RA-TTA computes, and $\mathcal{C}_y$ is a small set of database images the system retrieves confidently for class $y$. The denominator, the median cost over $\mathcal{C}_y$, fixes a per-class scale so that one threshold works across classes. A score well above 1 means even the best matches are atypically far from the query, which signals a hole. We synthesise only for the top few candidate classes whose score exceeds the threshold. At full coverage the pipeline therefore reduces to RA-TTA.
The second stage generates references along a path toward the target style. Classes are synthesised once and offline, not per query, so cost is paid per class. Each class is generated by image-to-image diffusion seeded with a handful of unlabeled target images from the test pool, and prompted with the class's descriptions. Varying the denoising strength $s$ traces a path. At low $s$ the output keeps the target appearance, such as satellite texture or aircraft-photo framing, but may not show the class. At high $s$ the output shows the class but drifts back toward the generator's natural-photo prior. This path is the pixel-space analogue of GOAT's stepping stones. It runs between "looks like the target domain" and "is clearly this class".
The third stage verifies each generated image and selects a strength. A candidate image is kept only if it passes a round trip. The first check is answer consistency: CLIP, scoring against the descriptions of the class and of its nearest confusable classes, must recover the intended class. The second check is description consistency: a captioning model's description of the image must be judged consistent with the class description by a text-only entailment check. The third check is a blind check in the spirit of VeriSciQA: the image must not be classifiable as the intended class from target-style features alone, which catches outputs that simply copy the seed image. For each class we pick the largest pass rate at the lowest strength. Passing images are added to $R_y$ only when holes are detected. Classes with near-zero pass rates are recorded as "generator also lacks this concept". For those classes the system falls back to description-only scoring rather than trusting noise.
Research program
We will build this on top of RA-TTA's released setting, with the same LAION2B sub-database, CLIP backbone and description sets. The main analysis is a coverage-deletion study. For each test dataset we remove from the database a fraction $\rho \in \{0, 0.5, 0.9, 1.0\}$ of the images that match each held-out class. An image counts as matching if its caption contains the class name or a synonym, or if its CLIP similarity to the class descriptions exceeds a threshold. Near-duplicates of test images are also removed. This yields accuracy-versus-coverage curves, which we summarise as the area under the curve.
We must state one confound openly. Stable Diffusion was trained on LAION, so deleting images from the index does not delete the concept from the generator. That is partly the realistic case, since a model's parametric memory is often broader than its index. But it means our gains could reflect leakage from the generator's training data rather than real hole-filling. We handle this by splitting results by generator competence, measured as each class's verification pass rate. We also report the subset where the generator fails, where the honest expected result is no gain and no harm.
Baselines
The naive reference point is zero-shot CLIP with description prompts. It uses no retrieval and no adaptation, and RA-TTA reports it at 66.05% average on the 13 transfer datasets. Any method that falls below it at $\rho = 1$ is actively harmed by its references.
The strongest existing method is RA-TTA, reported at 73.09% on the 13 transfer datasets and 63.47% on four ImageNet shift datasets. We rerun it ourselves on the deleted databases with its published hyperparameters. Our method uses the identical backbone, descriptions, retrieval budget $k$ and fusion rule, differing only in what fills $R_y$. Beating it means matching it within noise at $\rho = 0$, where gating should make the two systems identical, and exceeding it in accuracy at $\rho \ge 0.9$ on the extreme-shift datasets, RESISC45 and FGVC Aircraft.
A pure synthetic-support baseline in the style of SuS-X-LC is reported by RA-TTA at 70.89%. We will implement it ourselves as text-to-image references for every class, without gating, so that it uses the same generator, the same number of images per class and the same wall-clock synthesis budget as our method.
Three ablations isolate the key ingredients. The first is "naive fill": generate references for detected holes from text alone, with no style anchoring and no verification. The second is "style without verification". The third is "verification without style". The proposal's claim rests on the full system beating naive fill. If naive fill matches it, the contribution collapses to "add synthetic images", which SuS-X already covers. A fourth ablation replaces the gate with always-on synthesis, to measure what gating protects at high coverage. A fifth adds SDA-style low-strength diffusion projection of the query, to test whether matching inside the synthetic domain closes a remaining gap.
Why not just…?
Why not just use a bigger database or live web search? Larger indices shrink holes but do not remove them for long-tail classes. MOTOR shows that even a large, specialised index returns confidently wrong neighbours rather than nothing. The deletion study measures exactly the regime a bigger index cannot reach. It also shows whether our gate correctly does nothing when coverage is good, which is the case where a bigger index would win.
Why not just generate synthetic references for every class, as SuS-X does? This is the most serious threat. RA-TTA's numbers show text-only synthetic support underperforming real retrieval, 70.89% against 73.09%, at full coverage. That fits our hypothesis that unstyled synthetic references sit far from extreme target domains. It does not prove that our style anchoring and verification fix the problem. The naive-fill and always-on ablations settle the question. If verified, styled references are no better than SuS-X-style references once matched for compute, we will report that synthesis, not the mechanism, explains any hole recovery.
Why not just interpolate in CLIP embedding space, GOAT-style, instead of generating pixels? Embedding interpolation between retrieved neighbours is far cheaper. If the hole lies "between" covered classes, it may work. But it cannot add class evidence absent from every neighbour, and GOAT itself was only shown on unimodal benchmarks such as Rotated and Color MNIST. We include it as an extra baseline that replaces generation with Sinkhorn-geodesic interpolation between the query's features and the nearest covered references. If it matches pixel synthesis at $\rho = 0.5$ but not at $\rho = 1.0$, that pattern itself shows where real synthesis is necessary.
Why not just ask a large vision-language model directly? Frontier models may know rare classes, but the setting here is cheap test-time adaptation. RA-TTA runs at about 0.117 seconds per sample, and Auto-GDA found GPT-4o roughly ten times slower than an adapted small model for grounding checks. We will report one open large vision-language model as a reference point at its measured latency. We found no reported result on Lacuna for such a model on RA-TTA's benchmark suite.
Decisive checks
The proposal succeeds if, on RESISC45 and FGVC Aircraft at $\rho = 1.0$, the full system recovers at least half of the accuracy RA-TTA loses between $\rho = 0$ and $\rho = 1.0$. Naive fill must recover clearly less. At $\rho = 0$ the full system must stay within noise of RA-TTA. We will also report two diagnostics. The coverage score should detect deleted classes with high AUROC (the area under the curve trading off true and false detections) on known deletions. Offline synthesis plus gated lookup should keep per-sample latency near RA-TTA's. The decisive negative check is the generator-incompetent subset. If gains appear there, verification is passing hallucinations and the filter is broken. If they appear only where the generator demonstrably knows the class, the result is honest but should be read as moving knowledge from the generator's parameters into the retrieval index, not as creating new knowledge.
References
- RA-TTA: Retrieval-Augmented Test-Time Adaptation for Vision-Language Models
- MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
- ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation
- Bridging Semantic and Modality Gaps in Zero-Shot Captioning via Retrieval from Synthetic Data
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency
- VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- Generative Gradual Domain Adaptation with Optimal Transport
- Self-Training with Dynamic Weighting for Robust Gradual Domain Adaptation
- Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment