Language-Steered Zoom Inside One Transformer Pass

Detecting a tiny object in a large image, such as a kayak in a 4K coastal frame or a pedestrian covering 0.05% of a driving scene, forces a tradeoff. Downsample the image and the object vanishes. Process every high-resolution patch and compute explodes. In small specialized datasets, that second option also lets the model memorize the background. We propose a transformer that starts from a coarse view of the image together with a text description of the target. Partway through its layers, it decides which coarse regions to "open up": each chosen region is replaced by its finer-resolution sub-patches, which join the same token sequence. The text shapes that decision. The sub-patches stay tied to their parent region through a structured attention pattern, so the model gets a zoom hierarchy without a separate graph network or repeated re-encoding.

The claim is aimed at the data-scarce regime. If the model only ever spends high-resolution tokens on text-relevant regions, it sees far fewer background tokens during training. It should therefore overfit less to the fixed backgrounds that small specialized datasets contain, and it should generalize better to new sites. That hypothesis can be tested, and it is what separates this proposal from simply "adding multi-scale to a multimodal model."

Core Claim

A single-stream transformer processes text tokens and image-patch tokens together in one shared stack of self-attention layers. We claim that such a model can learn a coarse-to-fine expansion policy guided by the text, with parent–child containment built into its attention mask. At a matched budget of visual tokens, it will beat both exhaustive high-resolution tiling and iterative external zooming on small-object grounding. The advantage should grow as the training data shrinks and as the test background departs from the training background. The mechanism is new, and so is the prediction: text-conditioned sparsity in high-resolution tokens acts as a regularizer against background memorization.

Where This Sits in Prior Work

The closest work is Referring Expression Comprehension for Small Objects. It introduces the SOREC benchmark and PIZA, a 3.5M-parameter adapter on GroundingDINO. PIZA predicts a sequence of crop boxes autoregressively and re-encodes each crop through a dual-stream detector, meaning one with separate image and text encoders. PIZA establishes that zooming under language guidance works for tiny objects. We differ in three ways. Our zoom happens inside one forward pass. Coarse and fine tokens coexist and exchange information instead of the coarse view being discarded after cropping. Several regions can be expanded at once instead of following a single search path.

COST motivates the single-stream choice. It fuses text, template, and search tokens in one homogeneous encoder stack for small-object tracking and reports +3.1% AUC over TransT on VL-SOT230. However, COST works on a single patch scale. SIMLA constrains the design. Its evidence that raw visual tokens need dedicated vision-only layers before fusion with text (12 vision layers then 12 fused layers) is the reason our expansion step sits after a vision-only stage rather than at the input. The COST–SIMLA tension over homogeneous versus asymmetric stacks is one thing our ablations measure directly in the multi-scale setting.

The containment structure comes from computational pathology, where the hierarchy is built outside the transformer. PTree-Net selects regions on a thumbnail and builds a parent–child patch tree processed by a graph convolutional network. DAS-MIL and Ma et al. use cross-scale containment edges in graph attention networks over frozen features. MAPLE adds offline GPT-4 text prompts that filter patches before a cross-scale graph. All of these supply our structural prior: children must know which parent contains them. Shrivastava et al. found that flat multi-zoom concatenation often underperforms a single scale, which constrains us. We never treat scales as independent tokens.

ConSlide is the nearest hierarchical transformer, with bidirectional patch–region interaction. It is unimodal, and it has no text-driven selection.

Research Program

The model encodes a downsampled image into level-0 patch tokens. These pass through $L_v$ vision-only layers, following the SIMLA-style asymmetry, and are then concatenated with text tokens for the fused layers. At a designated fused layer, a small head scores every level-0 token. The top-$k$ tokens are expanded: the corresponding image region is re-patchified at twice the resolution, and its four child tokens are appended to the sequence. Each child carries three signals: a level embedding, its parent's 2D position, and a within-parent offset. The process can repeat for a second level. The expansion score for coarse token $i$ is

$$s_i = w^\top h_i^{(\ell)},$$

where $h_i^{(\ell)}$ is that token's hidden state at expansion layer $\ell$ and $w$ is a learned vector. Because $h_i^{(\ell)}$ has already attended to the text in the fused layers, the score is conditioned on the text without a separate module.

The attention mask lets each child token attend to the text, to all level-0 tokens (which supply global scene context), to its ancestors, and to its siblings. Children of unrelated parents are masked from one another. This mask replaces the external containment graph used in pathology models, and it bounds cost. The sequence grows by roughly $4k$ tokens per level instead of by the full high-resolution grid.

Two training signals drive the selection. The first is cheap: existing box labels mark which coarse cells contain the target, giving a supervised expansion loss. The second is the grounding loss, backpropagated through a straight-through top-$k$ estimator. No new annotation is required, which matters in the scarcity regime.

The vision layers are initialized from a pretrained ViT and the fused layers from a pretrained single-stream vision-language checkpoint. All baselines receive matched pretraining where feasible, because learning from scratch on scarce data would confound the comparison.

Baselines

The naive reference point is GroundingDINO fine-tuned on the downsampled image with no zooming. It shows how much is lost to the resolution bottleneck.

The strongest competitor on SOREC is PIZA-Adapter+ from Referring Expression Comprehension for Small Objects. It reports 36.8% mean accuracy on the SOREC validation set with 3.5M trainable parameters. For GroundingDINO full fine-tuning on SOREC, no reported result was found on Lacuna. PIZA is the strongest competitor because it already solves the same problem with language-guided zoom. We rerun it with the authors' configuration on our data splits and our data-fraction subsets. We use the same hyperparameter search budget for PIZA and for our model, and we report its total visual tokens encoded across zoom steps so that it can be compared at matched budget. Beating PIZA means exceeding 36.8% mAcc on the full validation set, or matching it with fewer visual tokens. More importantly, it means winning by a widening margin at 10% and 1% of the training data.

On RampWatch, the strongest reported system is YOLO-TG from RampWatch. It injects frozen Grounding DINO text features into YOLOv11 by cross-attention and reports mAP@50 of 0.71. Its cross-dataset results are reported only as relative gains over its own baseline: +22% on the Singapore Maritime Dataset and +4.3% on the Split Port Ship Classification Dataset. We retrain YOLO-TG on identical splits so that absolute transfer numbers can be compared.

Three ablations isolate the key ingredients. The first uses no text: the expansion score is computed before fusion, so selection is visual-only. It tests whether language steering matters. The second removes the containment mask and places children as ordinary tokens with absolute positions, which is the in-transformer analogue of flat concatenation. The third uses homogeneous early fusion, with no vision-only layers before expansion, to test the COST-versus-SIMLA question in the multi-scale setting.

Why not just…?

Why not just tile the high-resolution image and run a fine-tuned grounding detector on every tile? This is the most serious threat. With enough compute, exhaustive tiling sees every pixel. We treat it as a first-class baseline at two operating points: its natural full-tile budget and a budget capped to match ours. If full tiling equals our model on in-domain SOREC accuracy, the in-domain efficiency claim becomes only a compute claim. The proposal's substance would then rest on the scarcity and background-shift experiments, where tiling exposes the model to many more background tokens. If tiling also wins there, the regularization hypothesis is false and we will report it as such.

Why not just use PIZA's iterative zooming, which already works? PIZA discards the coarse context after each crop, and it follows a single search path, so an early wrong crop cannot be recovered. SOREC expressions average 25.5 words and often describe a target relative to its surroundings, which is exactly where losing context should hurt. We will split SOREC results by whether the expression mentions other objects. We will also measure PIZA's failure rate when its first zoom step misses the target, and compare it with our multi-region expansion.

Why not just filter patches with text first and then fuse scales with a graph network, as MAPLE does? MAPLE's text prompts are fixed offline and its patch selection is disconnected from the graph that reasons over the patches. It reports 74.0% AUC at 4-shot and 90.3% at 16-shot on TCGA-NSCLC. We will run our architecture on the same few-shot TCGA-NSCLC protocol as a secondary regime, with each magnification treated as one expansion level. If MAPLE's decoupled pipeline matches us there, the single-pass integration adds little in pathology, and we would restrict the claim to natural-image small objects.

Decisive Experiments

The central experiment trains every method on 100%, 10%, and 1% of SOREC and plots mAcc against training-set fraction and against visual tokens processed. We predict that our model's advantage over tiling and PIZA grows as data shrinks.

The background-shift check trains on RampWatch and tests zero-shot on the Singapore Maritime and Split Port datasets. It reports absolute mAP@50 and the in-domain-minus-transfer gap. We predict our model has the smallest gap. We also predict that the gap correlates with the fraction of background high-resolution tokens a method processes. We can vary that fraction within our own model by changing $k$, which gives a dose-response test of the regularization hypothesis rather than just a single comparison.

The no-text and no-mask ablations must each cost accuracy for the mechanism to be credited. If removing the text-conditioned score changes nothing, the result is a multi-scale detector and not a language-steered one.