Some visual questions can only be answered by looking something up: "What is the wingspan of this bird?" or "When was this tower built?" Systems that answer them first have to name the thing in the picture, and then retrieve facts about it from Wikipedia or Wikidata. The naming step is where they break. On INFOSEEK, a pipeline that names the entity with CLIP reaches 18.2% accuracy, while the same reader given the correct entity reaches 45.6% (INFOSEEK). On Encyclopedic VQA, automatic retrieval reaches 48.8% against an 87.0% oracle (Encyclopedic VQA). Those gaps were measured on images where the entity is usually large and centred. Real photographs often show the bird as a speck in a tree line, or the tower as a few dozen pixels on a skyline. In those images, naming fails for a second, compounding reason: there are not enough pixels left to recognize the object.
This note proposes to measure and fix entity linking as a function of how many pixels the target occupies. Two different failures hide under "small." In the first, the detail exists at native resolution, but the encoder's resize throws it away; zooming fixes this. In the second, the target is intrinsically tiny and no zoom recovers what the sensor never captured. Here the only remaining evidence is the surroundings: the habitat around the bird, the coastline behind the tower. We propose a linker that scores each candidate entity by two things. One is how well a zoomed crop matches it. The other is how well the surrounding scene agrees with what the knowledge base says about that candidate. A learned gate decides, per image, how much to trust each source based on how degraded the crop is.
Core Claim
Zoom-then-link cascades, which pair high-resolution crop retrieval with an entity linker, will close most of the gap when the problem is encoder-induced resolution loss. They will plateau when the target's native footprint falls below roughly the encoder's patch size. In that second regime, a context term grounded in external knowledge recovers a large share of the lost accuracy. The context term matches the target's neighbourhood against each candidate's encyclopedic description, for example habitat, location, or typical surroundings. It works only if its weight adapts to crop quality. Without that adaptation, it either does nothing (low weight) or makes the linker guess from co-occurrence on large, clearly visible targets (high weight).
Two features separate this from the closest work. EntRAG binds entity text to visual patch features and reranks candidates, but it runs at standard input resolution. It also notes that CLIP linkers lean on global image cues, which it treats as a nuisance rather than as evidence to be weighed. The high-resolution perception methods RAP and MRD retrieve crops from within the image but never consult an external knowledge base. Our proposal joins these two lines and adds the piece neither has: context evidence whose relational prior comes from the knowledge base, not from co-occurrence statistics learned in closed-world training.
Research Program
The first deliverable is an evaluation protocol rather than a model. We take the Encyclopedic VQA and INFOSEEK evaluation splits and locate the queried entity in each image with an open-vocabulary detector, as MRD does with LLMDet. Each question is then indexed by the entity's footprint in native pixels. The two regimes are created by two kinds of degradation. Encoder-induced degradation keeps the native image but places it within a larger canvas or feeds it at lower encoder resolution, so the detail survives in the file. Intrinsic degradation downsamples the whole image before any processing, so the target shrinks to a fixed footprint, such as 8, 16, or 32 pixels on a side, while its real surrounding scene is preserved. Downsampling the full image keeps the context natural. We avoid pasting targets into other scenes, except in one deliberate counterfactual test described below. We will also report results on the naturally small-footprint subset, without synthetic degradation, so that conclusions do not rest only on simulation.
The linker itself has three parts. First, a question-conditioned crop locator, which reuses RAP/MRD-style retrieval over image tiles, proposes the target region at native resolution. Second, the crop and a context ring around it are encoded by frozen backbones. The context ring is pooled on a log-polar grid, following HoughNet's observation that nearby evidence should keep fine spatial detail while distant evidence can be coarsened. Following ComBo, several backbones are combined (CLIP or SigLIP for semantics, DINOv2 for fine texture) through a small trained head. The backbones themselves are never fine-tuned, which keeps training cheap. Third, each candidate entity $e$ from the knowledge base is scored as
$$s(e) = g \cdot S_{\text{crop}}(c, e) + (1-g) \cdot S_{\text{ctx}}(r, e), \qquad g = \sigma\big(h(\phi(c), m, a)\big),$$
with the terms defined as follows. $S_{\text{crop}}$ is the similarity between the crop embedding $c$ and the candidate's reference images and name. $S_{\text{ctx}}$ is the similarity between the context-ring embedding $r$ and the candidate's knowledge-base description: its habitat or range sentences for species, its location and setting for landmarks. The gate $g$ is computed by a small network $h$ from three inputs: crop features $\phi(c)$, the margin $m$ between the crop's top two candidates, and the crop's native area $a$. The gate is trained on Encyclopedic VQA training data with randomized downsampling, so it sees every footprint. The top-ranked entities are passed to an unchanged reader, either the FiD-style reader used in INFOSEEK or the generator stage of EntRAG. Any gain can then be attributed to linking.
Baselines
The naive reference point is the standard pipeline: a CLIP entity linker applied to the whole image resized to encoder resolution, followed by the same reader. On INFOSEEK this is the CLIP + FiD system, reported at 18.2% accuracy (INFOSEEK). We rerun it on every footprint stratum to produce the degradation curve. This baseline already uses global context implicitly, which makes it an important control.
The strongest existing knowledge-seeking method is EntRAG, reported at 46.1 accuracy on Encyclopedic VQA and 44.5 on InfoSeek. We will run its released pipeline, or a faithful reimplementation if no release is available, on the same footprint strata with the same knowledge bases. The relevant comparison is the shape of the curve: EntRAG should match us on large targets and fall behind on small ones. No reported result was found on Lacuna for any method on a footprint-stratified or small-target split of either benchmark. All small-target numbers will therefore be new measurements, and we will report the full-split numbers alongside them to confirm our reruns match the published figures.
The strongest constructed competitor is a zoom-then-link cascade. It uses MRD's crop localization, which lifts LLaVA-v1.5-7B from 48.7% to 95.6% on V* Bench (MRD), then feeds the best native-resolution crop to the EntRAG linker. This is the baseline we expect to be hardest to beat in the encoder-induced regime. Beating it means a statistically reliable top-1 entity recall gain in the intrinsic regime at footprints of 16 pixels and below, with no loss above 64 pixels.
Two ablations isolate our ingredients. The first replaces the adaptive gate $g$ with a single fixed weight tuned on the development set. The second replaces the knowledge-grounded context term with the global image embedding, so that context is matched against candidate images rather than candidate descriptions. All systems share the same backbones, knowledge base, reader, crop locator, and a matched hyperparameter search budget. Latency per query is reported for every system.
Why not just…?
Why not just zoom in and link the crop? This is the most serious threat, and in the encoder-induced regime we expect it to be about as good as our method. Zooming cannot create pixels the sensor never recorded. GLAD's local high-resolution search raised appearance recall only from 0.17 to 0.51 for tiny drones (GLAD), which leaves many targets still unresolved by appearance alone. The two-regime protocol settles this question directly. If the cascade matches us in the intrinsic regime too, the context term adds nothing, and we will report that.
Why not just link with the whole image, since it already contains the context? Whole-image embeddings do carry scene information. EntRAG notes that this is exactly what CLIP linkers over-weight. Our concern is that the whole-image embedding mixes context with the target at a fixed ratio and matches it against reference photos, which may show the entity in a different setting. The ablation that swaps the knowledge-grounded context term for the global embedding tests whether matching against descriptions such as "found in coastal marshes" adds anything beyond image-to-image scene similarity.
Why not just use a fixed blend of crop and context scores? If one tuned weight works across footprints, the gate is unnecessary complexity. The literature gives reason to expect it will not. Static co-occurrence priors are known to hallucinate in atypical scenes; Direction 1 records detectors predicting a mouse because a keyboard is present. A weight high enough to help tiny targets should therefore hurt large ones. The fixed-weight ablation, evaluated across strata and on the counterfactual set described below, measures this directly.
Why not just hand the problem to a large agentic model with zoom and web search? Agentic systems such as MM-DeepResearch and GeoVista can in principle zoom, search, and reason about context. Closed-book scale alone clearly fails here: PaLI scores 13.0% on Encyclopedic VQA. We will run one open agentic baseline under a matched tool-call budget, on a subsample of the small-footprint strata. If it matches us, our contribution narrows to cost, and we will say so.
Proposed Experiments
The central experiment plots top-1 entity recall and VQA accuracy against native target footprint, separately for each regime, for every system above on Encyclopedic VQA and INFOSEEK. The prediction is specific. The zoom cascade should track our method in the encoder-induced regime and diverge from it below roughly 16 pixels in the intrinsic regime. The gap should be larger for species, whose habitat is described in the knowledge base, than for landmarks, whose surroundings vary with viewpoint. The size of that divergence, measured against the oracle-entity ceiling, is the headline result.
Two diagnostic checks would show whether the context term is doing what we claim. In the context-swap test, each target's context ring is replaced with the ring from another image of a different candidate entity. Our method's small-target accuracy should collapse toward the crop-only baseline, while its large-target accuracy should barely move. This would show both that context is being used and that the gate stops it from overriding a clear crop. In the incongruent-context test, a clearly visible target is pasted into a setting its knowledge-base entry contradicts, and the false-link rate is measured. The fixed-weight ablation should be fooled more often than the gated model.
Finally, we will check that nothing breaks on standard data by reporting full-split Encyclopedic VQA and INFOSEEK accuracy against EntRAG's 46.1 and 44.5. We will also report latency, given the known cost of multi-stage retrieval; REVEAL, for example, needs very large memory for its index.
References
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG
- MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
- HoughNet: Integrating Near and Long-Range Evidence for Visual Detection
- Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
- Global-Local MAV Detection Under Challenging Conditions Based on Appearance and Motion
- MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline
- GeoVista: Web-augmented Agentic Visual Reasoning
- Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory