Interactive image retrieval asks a user to steer a search engine with feedback such as "like this one, but with longer sleeves." It is one of the main claimed remedies for the semantic gap, which is the mismatch between what a model measures in pixels and what a person means. Systems in this line are almost never tested with real people. They are trained and scored against a user simulator: a model that looks at the hidden target image and at the system's current suggestion, then writes the feedback a person might give. That simulator sees the answer. If its feedback carries more about the target than a human's would, a retrieval system can score well without ever learning to interpret intent. Agent research has already met this failure in another form. Audits found that "do-nothing" agents beat strong models and that agents could overwrite the grading scripts. The fix there was to run evaluation in sealed, version-pinned containers with explicit checks on what each component may read and change.
We propose to bring that sealed-evaluation discipline to interactive image retrieval and to test one claim. A measurable share of the reported gains from conversational and composed retrieval comes from the evaluation setup: simulator feedback that leaks the target, missing trivial baselines, and unpinned random components such as diffusion-generated query images. The gains do not all come from the retrieval method narrowing the semantic gap. We will build a hermetic harness with an information-limited simulator and a leakage probe, re-run representative methods inside it, and report which gains survive.
Core Claim
The quantity that matters is how much of a method's improvement over a non-interactive baseline remains after two controls. The first control replaces the learned interaction policy with a "feedback-only" query. This query concatenates every simulator utterance from the session and issues it once to a frozen text-image embedding model, with no dialog manager and no candidate history. The second control caps how identifying simulator feedback can be, at the level measured from human-written feedback on the same image pairs. Write $R_{\text{method}}$ for the method's Recall@K after $T$ turns, $R_{\text{fb}}$ for the feedback-only baseline under the same utterances, and $R_0$ for single-shot retrieval from the initial query. We call $(R_{\text{method}} - R_{\text{fb}})/(R_{\text{method}} - R_0)$ the interaction share. It is the fraction of the method's improvement that the feedback text alone cannot explain.
Our hypothesis is that under current model-based simulators the interaction share is small, and that it grows once the simulator's leakage is capped at human levels. If that holds, the field has been partly measuring simulator quality rather than retrieval quality. If it fails and interaction share stays high under capped leakage, the harness gives these methods a stronger evaluation than they have now. Either outcome is useful.
Where the Two Directions Meet
The reproducible-execution literature provides the method. OpenHands runs agents inside Docker sandboxes so that every run starts from the same clean state. BOSS-LDG and BIDS apps show that shipping a full operating-system image makes the same binaries run unchanged on different clusters. We use this idea to pin the CUDA stack, model weights, and samplers of every retrieval component.
Agent-Diff contributes the "state-diff contract." It snapshots environment state before and after a run and checks that only intended changes occurred. We adapt it so that the gallery index, the target store, and the simulator's allowed inputs are snapshotted and checked after each session.
Establishing Best Practices for Building Rigorous Agentic Benchmarks is the main motivating paper. It reports that 70% of ten audited agent benchmarks violated both task and outcome validity. It also reports that an empty-response agent reached a 38% success rate on τ-bench, above GPT-4o. Its checklist is generic and contains no retrieval-specific failure modes. Simulator leakage is the retrieval counterpart of an agent editing its own grader, and the checklist would not catch it.
SciAgent's audit harness adds append-only provenance logs and a "fresh-context verifier," an LLM that sees only logs and outputs. We reuse the logs. We treat its verifier only as a secondary check, because the same paper reports that such verifiers are uncalibrated and can hallucinate verifications.
The retrieval literature provides the setting and the reasons for concern. Dialog-based Interactive Image Retrieval trains a reinforcement-learning dialog manager against a "relative captioner" simulator that describes the difference between candidate and target. The Fashion IQ Dataset supplies human relative captions and uses multimodal transformers both to retrieve and to simulate users. Both papers note that the simulators are domain-bound and costly to supervise. Neither measures how much target identity a simulated caption carries compared with a human one, and that is the gap we fill.
ADaFuSE adds diffusion-generated proxy images to interactive text-to-image retrieval. It reports that naive additive fusion makes retrieval worse in over 55% of samples. That result shows a stochastic generator can dominate outcomes, so seed and host variance must be measured before differences between methods are trusted.
RUNC models uncertainty in composed retrieval, and Cola shows that off-the-shelf CLIP and FLAVA fall below the 25% chance level on compositional distractors. Cola matters to us as a constraint. The feedback-only baseline uses a frozen embedding model, and that model is weak at binding attributes to objects. A low feedback-only score could therefore mean either "little leakage" or "the embedder cannot parse the leaked detail." We handle this by also running the feedback-only probe with the Cola-style adapter (MM-Adapter) and with a reranking model.
Finally, Content-based Image Retrieval and the Semantic Gap in the Deep Learning Era shows that gains in instance retrieval do not transfer to semantic retrieval: mAP on Revisited Oxford rose from about 0.3 to above 0.65 while semantic retrieval on MIRFLICKR-25K stayed flat. It also criticizes binary relevance metrics. That paper is the reason we ask whether the semantic gap is actually narrowing rather than whether recall numbers rise.
Research Program
The first component is the harness. Every component of an interactive retrieval run is built as a pinned container image with content hashes on its weights: the gallery index, the retrieval method, the simulator, and any generator. Components talk only through a logged message bus. The simulator container can read the target image and the current candidate and can write only text. The retrieval container can never read the target store. A state-diff contract runs after each session. It asserts that the gallery index is byte-identical to its pre-session snapshot, so no target can be injected or reweighted through side channels. It also asserts that the target store was accessed only by the simulator and that no outputs from the grading process reached the retrieval container. Each method is run under a grid of seeds and on at least two GPU types. This yields a cross-host variance band for every reported number, in the same spirit as the multi-cluster reproducibility work.
The second component is the leakage probe and the leak-capped simulator. Leakage is measured as the retrieval rank of the target when only the feedback text, with no candidate image, is used as a query, averaged over a fixed probe ensemble: frozen CLIP, CLIP with MM-Adapter, and a vision-language reranker over the top-100. We first measure this on Fashion IQ's human relative captions to set a human leakage profile for each category. We then measure the same profile for model-based simulators: the original relative captioners and modern vision-language models prompted to give feedback. To build a leak-capped simulator, we reject and resample simulator utterances whose probe rank is better than a chosen percentile of the human profile, with a fixed sampling budget. This gives a family of simulators running from "human-like leakage" to "unconstrained." We can then plot each method's recall against simulator leakage instead of reporting a single number.
The third component is the re-evaluation. Representative methods are run inside the harness against every simulator setting. The methods are a learned dialog manager in the style of Dialog-based retrieval, a Fashion IQ multimodal transformer, a generative-proxy method with adaptive fusion in the style of ADaFuSE, and an uncertainty-calibrated composed retriever in the style of RUNC. For every cell we report Recall@K, the target's percentile rank, interaction share, and the cross-host variance band. Because the semantic-gap literature distrusts binary metrics, we also report graded relevance on Fashion IQ, using attribute overlap with the target, so that near-misses are distinguished from unrelated results.
Baselines
The naive reference point is single-shot retrieval from the initial query with a frozen embedding model and no feedback at all. This is $R_0$. It is the image-retrieval counterpart of the empty-response agent, and if it is close to the interactive methods, the benchmark is not measuring interaction. It runs in the same container with the same gallery and the same probe models.
The key ablation is the feedback-only baseline described above. It isolates whether the learned interaction policy adds anything beyond the information the simulator emitted. It receives exactly the utterances the method's own session produced, replayed from the provenance log. It therefore uses identical information at zero training cost, and it is given the same reranker budget as the strongest method. A method beats this baseline only if its advantage exceeds the cross-host variance band at every simulator leakage level, and especially at the human-matched level.
The strongest existing methods are the reinforcement-learning dialog manager of Dialog-based Interactive Image Retrieval and the generative-proxy fusion of ADaFuSE. They are the strongest competitors because they are the methods most explicitly claiming that multi-turn interaction narrows the gap. Each is retrained or fine-tuned with its published recipe under an equal GPU-hour budget and an equal hyperparameter-search budget, against each simulator setting. Re-tuning against the leak-capped simulator is therefore a fair test and not a handicap. We found no reported Recall@K for these methods on Fashion IQ or the shoes dialog benchmark on Lacuna, so we will reproduce their numbers in the harness and report those reproductions next to the published claims. The only nearby reported figures in the file are Cola's finding that off-the-shelf CLIP and FLAVA score below 25% chance on compositional distractors, which bounds what the frozen-embedder probe can detect, and the ADaFuSE observation that additive fusion degrades over 55% of samples.
Why not just…?
Why not just apply the Agentic Benchmark Checklist to these retrieval benchmarks? The checklist asks whether outcomes are graded correctly and whether trivial agents are tested. It has no item for a grader-side component, the simulator, that holds privileged information and passes it to the system under test. Applying it would likely mark interactive retrieval benchmarks as valid, because the grader (rank of the target) is correct. The flaw sits upstream, in what the simulator says. We will still score each benchmark against the checklist, so readers can see what it catches and what the leakage probe adds.
Why not just evaluate with real humans? Human studies are the gold standard for intent, but they are expensive, slow, and impossible to repeat exactly, and the Fashion IQ and dialog-retrieval authors already cite collection cost as a limitation. We use humans where they are cheapest and most informative: their existing relative captions set the leakage ceiling. As a validity check, we run a small live study with a fixed set of target images. It tests whether method rankings at the human-matched simulator setting agree with rankings from real users better than rankings at the unconstrained setting do. If they do not, the leak-capped simulator is not a valid stand-in, and we will say so.
Why not just fix random seeds and publish a requirements file? Seeds do not control nondeterministic GPU kernels, driver differences, or changes in model weights served behind APIs. The reproducibility literature documents this kind of drift. The ABC audit, for example, records OSWorld evaluation breaking because live web content changed. Our cross-host runs test this directly. If seed-only variance equals the cross-host band, the container layer adds nothing for this problem and we will drop that claim. We expect generative-proxy methods to show a wider band, given ADaFuSE's sensitivity to proxy quality.
Why not just use a stronger vision-language model as the simulator? A stronger simulator writes more fluent, more specific feedback, which is the direction that increases leakage. It may make the benchmark easier for the wrong reason. Our leakage-versus-recall curves test this: if stronger simulators move every method up and also raise feedback-only recall by the same amount, interaction share falls and the strength of the simulator is the confound.
Decisive Checks
Three results would settle the claim. The first is whether feedback-only recall under the unconstrained model simulators comes within the cross-host variance band of the strongest method on Fashion IQ. If it does, the reported interaction gains are explained by leakage. The second is whether interaction share rises as the simulator is capped toward human leakage. That would show the methods do use interaction, but that current simulators hide this behind leaked detail. The third is whether rankings between methods hold across hosts and seeds. If differences between methods fall inside the variance band, current leaderboards cannot separate them.
As a secondary check tied directly to the semantic gap, we test whether methods that gain on Fashion IQ under capped leakage also gain on graded relevance and on MIRFLICKR-25K-style concept queries. The Barz and Denzler result predicts that specificity gains may not transfer to abstract intent. All containers, simulator settings, provenance logs, and probe models will be released so that any new interactive retrieval method can be placed on the same leakage-versus-recall curve.
References
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- BOSS-LDG: A Novel Computational Framework That Brings Together Blue Waters, Open Science Grid, Shifter and the LIGO Data Grid to Accelerate Gravitational Wave Discovery
- BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods
- Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- Audit-Grade Harness for Agent-driven Scientific Computation Workflows
- Dialog-based Interactive Image Retrieval
- The Fashion IQ Dataset: Retrieving Images by Combining Side Information and Relative Natural Language Feedback
- ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval
- Towards Robust Uncertainty Calibration for Composed Image Retrieval
- Cola: A Benchmark for Compositional Text-to-image Retrieval
- Content-based Image Retrieval and the Semantic Gap in the Deep Learning Era