Visual search agents are vision-language models that answer questions about large images by repeatedly choosing a region, zooming in, looking again, and deciding where to go next. They now beat single-pass models on high-resolution benchmarks, but they fail in a characteristic way: they get stuck. Mini-o3 reports that its agent often zooms in and out of the same area and never returns to explore elsewhere. Visual CoT Makes VLMs Smarter but More Fragile shows that these multi-step pipelines lose more accuracy under corruption than single-pass models, because one bad intermediate crop poisons everything after it. We propose that much of this trapping has a specific, measurable cause. The agent is drawn toward regions whose content sits close to the target in its own embedding space, the same "crowding" that makes adversarial errors land on semantically neighboring classes.
In two sentences: search agents waste their turn budget on decoys that look like the target to their own encoder, and the waste grows with how close the decoy sits to the target in that space, not with how many distractors the image contains. This predicts a cheap attack: paste natural patches of a crowded neighbor class. It also predicts a cheap fix: when the agent rejects a candidate, it should mark that rejection in embedding space, so that every region resembling the rejected one is also deprioritized, rather than recording only the rejected coordinates. Graph-path complexity results tell us which version of the planning problem to avoid, namely insisting on a globally optimal non-repeating inspection order, and they give us a precise way to measure trapping.
Core Claim
The claim has three parts, and each can be falsified on its own. First, on a fixed image with a fixed number of inserted distractors, the turns an agent spends before finding the target, and its failure rate, rise monotonically with the embedding similarity between distractor and target. Raw distractor count, at fixed similarity, matters much less. Second, this makes a "crowded-decoy" attack: natural patches drawn from a semantic neighbor of the target class degrade search accuracy more than pixel-level perturbations of matched visibility. The resulting failures appear as long trajectories that revisit regions instead of short wrong answers. Third, a training-free controller that combines rejection marking in embedding space with explicit removal of explored regions from a region hierarchy recovers most of the lost accuracy. Coordinate-only memory and extra turns do not.
The first part transfers a known static result to a sequential setting. In static classifiers, untargeted attacks pull predictions toward nearby false classes, and on ImageNet most transferred adversarial examples land in classes already in the clean top-5 (the result attributed to Pintor et al. in the file). TIMA shows for CLIP that semantic proximity between classes predicts adversarial fragility. Both concern a single forward pass. Our claim is about where a multi-turn policy spends its budget, and nobody has measured that. Decoding in Geometry is the closest methodological relative. It shows that probability mass collapsing onto clusters of nearby token embeddings predicts reasoning errors, and its training-free method downweights such clusters during decoding. We borrow its idea of geometry-aware downweighting but apply it to choosing which crop to inspect next, using rejection evidence rather than a sampling correction.
Research Program
The first component is the measurement apparatus. We represent each image as a region hierarchy, a quadtree whose nodes are crops at increasing zoom, and log every agent trajectory as a walk on that tree. We define a trap, following the "impregnable subdigraph" notion of Kerivin et al., as a set of nodes the walk keeps cycling within. Every exit from that set leads to either another member of the set or the root without any new subtree being opened. Trap mass is the fraction of turns spent inside such sets. We also record revisit rate, meaning repeated entries to the same node, and the non-simplicity of the walk. Crowding is measured in the agent's own vision encoder, as the cosine similarity between the embedding of the decoy crop and the embedding of the target crop. We also compute a text-side version, the similarity of the decoy crop to the query.
The second component is the decoy benchmark. Starting from VisualProbe-Hard and V* Bench images, we insert natural object patches at controlled positions and scales. The patches come from five similarity bins, ranging from same fine-grained superclass (another breed, a near-identical icon) to unrelated. Distractor count is held fixed within each comparison. On ScreenSpot-Pro we do the same with visually similar but wrong interface icons, which are the domain's natural decoys. A gradient variant optimizes a small, low-amplitude perturbation confined to a far-away region so that its crop embedding moves toward the target. This tests whether crowding can be manufactured, as opposed to merely found.
The third component is the controller. It wraps an existing agent, Mini-o3 or Qwen2.5-VL-7B, without retraining. When the agent zooms in and concludes "not the target," the controller does two things. It deletes that subtree from the residual hierarchy. It also adds the rejected crop's embedding to a rejection set, so that every unexplored region whose embedding is within a calibrated radius of a rejected one has its proposal priority lowered rather than removed. Next-region selection then becomes a best-first choice over the remaining frontier of the hierarchy, and this is polynomial per step.
We deliberately do not ask for a globally optimal inspection order without revisits. Formal language constrained path problems shows that requiring simple paths under even fixed regular constraints is NP-hard on grid graphs, whereas walks that may repeat vertices remain polynomial through a product with a finite automaton. A small automaton handles phase constraints, for example "after rejecting a candidate, return to the parent before descending again." It runs on the product with the hierarchy, as in that paper's construction, so planning stays cheap. On small instances we report the optimality gap against exhaustive search, so the cost of avoiding the hard problem is measured rather than assumed.
The main risk is the tension the file identifies. Pushing apart crowded representations costs clean accuracy: Robustness May Be at Odds with Fairness reports average accuracy falling from 78.9% to 75% under class reweighting. Our rejection marking could likewise suppress the true target when it resembles a rejected decoy. We therefore lower priority instead of deleting, always permit full-zoom verification of any region, and report false-suppression rates directly.
Baselines
The naive reference point is the backbone, Qwen2.5-VL-7B, answering in a single pass at its native maximum resolution, alongside the same model with a fixed 4-turn budget. This tells us how much of any effect belongs to multi-turn search at all.
The strongest existing method on our core benchmark is Mini-o3 run at 32 turns. Mini-o3 reports 48.0% accuracy on VisualProbe-Hard at 32 turns, against about 25% at 4 turns and 35.1% for DeepEyes, along with 88.2% on V* Bench. It is the right competitor because it is the system that documented the backtracking failure, and its own answer to that failure is to scale up turns. We run it with the released weights, on the same clean and decoy images, at 32 turns. Our wrapped version gets the same turn cap, so any gain cannot come from extra compute. Beating it means higher accuracy on the high-crowding decoy bins at equal turns, with no statistically significant loss on clean VisualProbe-Hard.
For multi-hop chart and map search, the strongest reported system is the InSight-o3 searcher paired with Gemini-2.5-Flash, at 69.7% on O3-BENCH (60.4% without the searcher). For GUI grounding it is ScreenSeekeR from ScreenSpot-Pro, at 48.1% versus 18.9% for OS-Atlas-7B alone. We wrap each system's searcher with our controller, keep the reasoner and planner calls identical, and count model calls rather than wall time.
The detector baseline replaces the agent's own box proposals with Grounding DINO proposals. Visual CoT Makes VLMs Smarter but More Fragile used this swap and reported an average gain of about 6% under perturbation, exceeding 10% on DocVQA. The coordinate-memory baseline forbids re-entering rejected boxes but keeps no record of what they looked like.
Two ablations isolate our key ingredient: one controller with only subtree deletion and one with only rejection marking in embedding space. The central prediction is that rejection marking accounts for most of the recovery in the high-crowding bins and almost none in the unrelated-decoy bin.
Why not just…?
Why not just give the agent a memory of visited coordinates? This is the most serious alternative, and it is the coordinate-memory baseline above. It would fix literal loops of the kind Mini-o3 describes. Our hypothesis is that crowded decoys do not mainly cause literal loops. They cause the agent to visit a sequence of distinct regions that all resemble the target, and coordinate memory does nothing about that. If coordinate memory matches the full controller on the high-crowding bins, then trapping is about memory, not geometry. In that case we would drop the embedding-space component and report the simpler fix.
Why not just scale turns further, as Mini-o3 does? More turns help on clean images, from about 25% to 48.0% on VisualProbe-Hard. But if each extra turn is spent on another crowded decoy, the accuracy-versus-turns curve should flatten in the high-crowding bins while continuing to rise in the low-crowding bins. We plot exactly these curves up to 32 turns. If they do not flatten, the attack is weaker than claimed.
Why not just use an external detector such as Grounding DINO? A detector trained on categories will propose the crowded decoys too, since they are real objects of a neighboring class. In addition, InSight-o3 reports that standard detectors fail on relational queries. The detector baseline tests this directly on our bins.
Why not just adversarially train the backbone? Robustness May Be at Odds with Fairness shows that standard adversarial training widens the gaps between crowded classes. It also does not target natural decoys, which are not small perturbations, and retraining a 7B agent with adversarial rollouts is far outside a training-free budget. We therefore treat it as out of scope, stated rather than tested.
Proposed Experiments
The decisive first experiment is the dose-response test. On VisualProbe-Hard and V* Bench images carrying decoys at five crowding levels with the number of decoys held fixed, we regress success and turns-to-target on crowding and on distractor count. The claim fails if crowding has no effect beyond count, or if the trend reverses when crowding is measured in text space instead of image space.
The second experiment compares attacks. Crowded natural decoys go head to head with the region-confined gradient perturbation and with full-image PGD at matched visibility, measured by structural similarity to the clean image. We report accuracy, trap mass, and revisit rate against Mini-o3 and the InSight-o3 searcher. Under the theory, crowded decoys should raise trap mass, whereas random corruption should mainly raise immediate wrong answers.
The third experiment is the defense comparison. The full controller, its two ablations, coordinate memory, and the Grounding DINO swap are run at equal turn budgets on clean and decoy versions of VisualProbe-Hard, V* Bench, O3-BENCH, and ScreenSpot-Pro. The pass criterion is clean accuracy within noise of the reported numbers (48.0%, 88.2%, 69.7%, 48.1%) together with a clear recovery in the high-crowding bins. We also report the target false-suppression rate, the optimality gap against exhaustive search on small hierarchies, and, as an external check on whether trapping resembles human distraction, scanpath agreement on ViSioNS image sets where decoys can be inserted. That last comparison is a secondary analysis, not a pass criterion.
References
- Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search
- Visual CoT Makes VLMs Smarter but More Fragile
- TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability
- Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoning
- On the complexity of the Eulerian closed walk with precedence path constraints problem
- Formal language constrained path problems
- Robustness May Be at Odds with Fairness: An Empirical Study on Class-wise Accuracy
- InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
- ViSioNS: Visual Search in Natural Scenes Benchmark