When a vision-language model is asked to find "the cyclist in a yellow jacket behind the bus," it must do one of two things: draw a box around that cyclist, or say that no such cyclist exists. Models trained to say "not there" use one cue for that decision: the image evidence matching the phrase is weak. A real target that covers fifteen pixels produces the same weak evidence once the network's early downsampling has averaged it into one coarse feature. The model therefore cannot tell an absent object from a present but tiny one. Training it to reject absent targets teaches it to discard small real ones, and training it not to miss small objects teaches it to hallucinate on absent ones. No existing benchmark measures this trade-off. Null-target benchmarks such as gRefCOCO and OmniLabel use ordinary COCO-scale objects, and the one tiny-object referring benchmark, SOREC, contains no absent targets.
We propose to make the language query decide which fine image detail survives downsampling, and to make that decision before most of the network's depth is spent. The model is "one-tower": image patches and words share one transformer from the first layer, so the phrase can mark small high-resolution patches as worth keeping. The model is then trained on negatives built specifically at tiny scale. The claim is that absence becomes recognizable only when the evidence that would confirm a tiny target has not already been destroyed. We will test this claim by sweeping target size at a fixed compute budget.
Core Claim
In a referring model with early fusion, query-conditioned retention of fine patches combined with tiny-scale negative training lowers the rate at which present tiny targets are wrongly rejected. It does so without raising false alarms on absent targets, and it beats two simpler fixes at matched compute: uniformly higher resolution, and an existence head added to a standard model. The new elements are the retention mechanism, which conditions on the text, the negative data, which is built at tiny scale, and a diagnostic that separates "missed because too small" from "rejected because absent." We do not claim that early fusion alone helps.
Positioning Against Prior Work
Three works supply the absence-aware machinery we build on. GRES and its extension GREx introduce gRefCOCO and ReLA, a model that adds a no-target head on top of late cross-attention between region and language features. IR-VG adds a binary existence head and four stages of iterative vision-language fusion, trained against altered false descriptions. All three operate on standard backbone feature maps. They therefore inherit the problem we target: by the time their existence head runs, a sub-patch target is already blurred.
OneRef supplies our backbone design. It is a BEiT-3 one-tower model pre-trained with referring-aware masked image and language modeling, and it runs about ten times faster than models with heavy fusion decoders. OneRef constrains the proposal in two ways. It assumes every phrase has a target, and its fixed patch projection blurs anything smaller than a patch before any attention happens.
LIP supplies the retention idea. A small network predicts an importance map that weights pooling, which raises small-object AP on COCO. Its importance map is purely visual, so it keeps whatever is visually salient rather than what the query asks about. Effective Fusion Factor in FPN motivates our caution about mixing scales. It shows that deep, coarse features injected into shallow ones at equal weight hurt tiny-object detection.
Referring Expression Comprehension for Small Objects provides SOREC and PIZA. PIZA is an adapter that predicts a sequence of shrinking boxes, re-processing a crop at each step until an end-of-search token. It handles resolution through repeated passes, and it never has to decide that nothing is there.
Three further works constrain our negative data. Generating Enhanced Negatives shows that hard negatives from language-model text edits and diffusion inpainting help GLIP and FIBER on OmniLabel. It also reports that inpainting produces artifacts requiring heavy filtering and cannot work at pixel scale, so we do not use diffusion for tiny negatives. The study of context traps by He et al. finds that models answer from co-occurring scene context when an object is removed. That is precisely the failure our removal negatives must provoke. Radman observes that dense segmentation decoders are biased toward producing some mask, which argues for an explicit existence output rather than relying on a thresholded mask.
Research Program
The model. The model starts from a OneRef-style shared transformer. The image is tokenized twice: a standard 16-pixel patch grid, and a fine grid of 4-pixel patches embedded by a light convolutional stem. For the first $L_0$ layers (we plan $L_0 = 2$), the text and coarse tokens attend jointly, and the text also cross-attends to the fine tokens. A small scoring network then gives each fine token $i$ a keep score:
$$s_i = \sigma\big(g(f_i, \bar{t})\big),$$
where $f_i$ is the fine token's feature, $\bar{t}$ is the pooled text representation after $L_0$ fused layers, $g$ is a two-layer MLP, and $\sigma$ is the sigmoid.
The top $K$ fine tokens by score pass into the remaining layers alongside the coarse grid. All other fine tokens are folded into their parent coarse token by score-weighted averaging, which is the LIP rule with a text-conditioned importance. $K$ is fixed per run, so compute is controlled exactly. Selection is made differentiable with a straight-through top-$K$ estimator.
On top of the shared encoder sit two heads: OneRef's lightweight box/mask head, and an existence head that reads a dedicated null token. The null token attends over the final tokens, retained fine tokens included. The training loss sums the box/mask loss, applied only on positives, with a binary cross-entropy existence loss. We add one auxiliary term: on positives, the ground-truth box should overlap the retained fine tokens. This keeps retention from collapsing toward visually salient clutter.
The data. We build SOREC-Null from SOREC and the underlying SODA-D images, using three negative types.
Removal negatives replace the target's few pixels with background copied from a nearby region of the same image. At this scale, simple patch copying is visually clean in a way diffusion inpainting is not. These negatives leave the scene context intact, which is the context-trap condition.
Attribute foils use a language model to change one attribute, relation, or category word in the expression so that it describes a visible distractor incorrectly. This follows the generation recipe of Generating Enhanced Negatives. We verify each foil against SODA-D annotations so that it truly matches nothing.
Scale-matched positives are SOREC positives downsampled until the target falls below 8 pixels. They exist so the model learns that very small is not the same as absent, and they are paired with removal negatives at the same scale.
A human audit of a random sample will estimate the label noise in each negative type before any training.
The diagnostic. We place each test pair in a target-size bin by the target's pixel size after resizing to the model's input. In each bin we report three quantities: the miss-as-absent rate, which is the fraction of present targets the existence head rejects; the false-alarm rate on absent targets; and box accuracy on accepted positives. To control size while holding content fixed, we also rescale the same image–expression pairs to several resolutions. This includes gRefCOCO no-target and positive pairs shrunk so that COCO objects become tiny. The hypothesis predicts a specific result. For baselines, miss-as-absent should rise sharply as targets shrink while false alarms stay flat. For our model, the miss-as-absent curve should stay flat further down the size axis at equal false-alarm rate.
Baselines
The naive reference point is OneRef fine-tuned on SOREC with no existence head, plus a threshold on box confidence used as the absence decision. This measures how far confidence alone goes. On SOREC it should have reasonable positive accuracy and poor absence behavior.
The most important ablation is the same OneRef-style model with the existence head, trained on the same SOREC-Null data, with fine-token retention removed. It sees the same negatives and runs at the same compute. To match compute, the tokens freed by dropping the fine grid are spent instead on a uniformly finer coarse grid; see the resolution question below. This baseline isolates the claim. If it matches our model, the gain comes from the data, not the mechanism. We add a second ablation in which retention scores come from visual features alone, as in LIP, with no text input. Beating it shows that conditioning retention on the query matters, not just retention.
The strongest tiny-object positive-only competitor is PIZA. Its reported SOREC result is a validation mAcc of 36.8% with 3.5M trainable parameters (Referring Expression Comprehension for Small Objects). We will rerun PIZA from the same pretrained grounding backbone. We extend it with an existence head on the full-image pass, and we also let its end-of-search step emit "absent." It is trained on SOREC-Null with the same data and an equal hyperparameter-search budget. We report its accuracy and wall-clock cost per image. Beating it means matching or exceeding its positive mAcc on SOREC while achieving a lower miss-as-absent rate at equal false-alarm rate on SOREC-Null, with fewer forward passes.
The strongest null-target competitor is ReLA, with reported gRefCOCO validation gIoU of 63.60 in GRES and 63.98 in GREx. We will retrain ReLA on SOREC-Null and evaluate our model on gRefCOCO to check that the tiny-scale design does not cost performance at ordinary scale. Our target there is to stay within about one gIoU point of ReLA. We would also like to beat it on the rescaled-gRefCOCO bins.
On the reasoning side, Rex-Thinker reports a rejection score of 68.2 on HumanRef, against 7.1 for base Qwen2.5-VL, at 6.68 seconds per image (Rex-Thinker). We will run it zero-shot on SOREC-Null as a reference for accuracy and latency.
We found no reported result on Lacuna for any null-target method on SOREC or any other tiny-object referring benchmark. Every number on SOREC-Null will therefore come from our own reruns under matched conditions.
Why not just…?
Why not just feed the model a higher-resolution image? This is the most serious threat. Doubling input resolution also keeps tiny targets alive, at roughly four times the token count. We address it by giving the uniform-resolution baseline the same total token budget our model uses, with coarse grid plus $K$ fine tokens counted together. We also sweep budgets to trace accuracy against FLOPs for both. If uniform resolution matches us at equal FLOPs, the mechanism is not worth having, and we will report that. Our expectation that it will not match rests on SOREC targets averaging about 0.05% of the image area: a query-selected few hundred fine tokens can cover the target, while a uniform increase spends almost all of its budget on background.
Why not just add an existence head to PIZA and let zooming handle resolution? Zooming needs a first, coarse box. When the target is absent or already blurred, that first box lands on a distractor, and later zooms then refine a wrong region. The context-trap results of He et al. suggest the model will then confirm the distractor. Zooming also multiplies forward passes by the number of steps. The extended-PIZA baseline above tests the accuracy argument directly, so this claim is settled by experiment rather than asserted.
Why not just detect then reason, as Rex-Thinker does? Its reasoning stage can only judge boxes the Grounding DINO detector proposes, and the authors note that proposals are imprecise or missing in dense scenes. A tiny target the detector never proposes cannot be recovered, so the pipeline's miss-as-absent rate is bounded by the detector's recall. We will measure detector recall per size bin on SOREC to show this bound concretely. Latency, at 6.68 seconds versus about 1.13 for direct models, is a second cost.
Why not just train an existing model on hard negatives? Generating Enhanced Negatives shows this gives modest gains on ordinary scales: +2.9 AP for GLIP-T and +2.4 AP for FIBER-B on OmniLabel. Our hypothesis predicts that at tiny scale, negatives alone push the model toward rejecting real tiny targets, because the model has no feature that separates the two cases. The no-retention ablation trained on identical SOREC-Null data tests exactly this. Its miss-as-absent curve against target size is the decisive comparison.
Decisive Experiments
The main experiment trains all models on SOREC plus SOREC-Null and tests on a held-out SOREC-Null split. We report positive mAcc, existence accuracy, and false-alarm rate, and we trace the curve of miss-as-absent against false-alarm rate per size bin as the existence threshold varies.
The proposal succeeds if three things hold. First, at matched FLOPs, our model's miss-as-absent rate on targets under 16 pixels is clearly below both the no-retention ablation and the uniform-resolution baseline at the same false-alarm rate. Second, text-conditioned retention beats visual-only retention specifically on attribute foils, where the query, not visual salience, determines which pixels matter. Third, ordinary-scale gRefCOCO performance stays close to ReLA's reported 63.60–63.98 gIoU.
For transfer, we evaluate on OmniLabel's negative descriptions, comparing against the released enhanced-negatives GLIP checkpoint from Gen-Enhanced-Negs. We also check whether the retained fine tokens actually cover the ground-truth target on positives and stay scattered on removal negatives. This shows whether the model's absence decision reflects having looked at the right pixels and found nothing.
If the uniform-resolution baseline closes the gap at equal compute, or if the no-retention ablation closes it, we will report that negatives or resolution alone suffice. The size-binned diagnostic and SOREC-Null remain useful even in that case.
References
- GRES: Generalized Referring Expression Segmentation
- GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation
- Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision
- OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
- LIP: Local Importance-Based Pooling
- Effective Fusion Factor in FPN for Tiny Object Detection
- Referring Expression Comprehension for Small Objects
- Generating Enhanced Negatives for Training Language-Based Object Detectors
- Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects?
- TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation
- Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning