Models that find an object in an image from a sentence, such as "the dog on the left" or "the second woman from the right holding an umbrella," run every input through the same deep stack of layers. The strongest of these models use bidirectional cross-modal attention: the words look at image regions, and the image regions look back at the words, over several rounds. A solitary dog on grass gets the same number of rounds as a crowded street scene with a relational description. Multi-exit networks attach small prediction heads to intermediate layers so that easy inputs can stop early. They are an obvious fix, but the usual rule for when to stop, "the head is confident," is dangerous in this setting. A grounding model can be confident because the sentence alone suggests a typical location, while it has misread the image.
This note proposes early-exit grounding in which the stopping signal comes from the two attention directions themselves. At a given layer, the model is allowed to stop only when three things coincide. The words' attention over the image must concentrate on the box the intermediate head predicts. The image regions inside that box must attend back to the same words. That round trip must return to where it started. The claim is that this bidirectional agreement catches premature exits that confidence and layer-to-layer stability miss. If so, it gives a better accuracy-versus-compute curve on RefCOCO-family benchmarks. If it does not beat a simple stability rule, the experiments are built to show that plainly.
Core Claim and What Is New
Early exiting has been studied for text classification (The Right Tool for the Job, BERT Loses Patience, FastBERT), for document classification with a fused multimodal transformer (Multimodal Adaptive Inference for Document Image Classification), and for robot action regression (DeeR-VLA). In each case the stopping rule looks only at the output: softmax confidence, prediction persistence across layers, or a small change between consecutive predictions.
Three multimodal-LLM papers sit closest to the attention side, and all treat vision as a donor that can be switched off. DyVTE uses a trained MLP to pick a layer at which all visual tokens are dropped. HiDrop drops them at a fixed layer 25 and uses inter-layer attention similarity only to guide pruning. DeeR-VLA never touches the vision encoder at all.
None of these works does dense spatial grounding. None uses an exit rule that reads the agreement between the two attention directions. We do both. The grounding setting is not incidental. Here the prediction, a box, lives in the same spatial coordinates as the attention maps, so it can be checked directly against them. In classification no such check exists.
Mechanism
We start from a stacked co-attention grounding model in the style of PFOS. That model has two paths: a language-guides-vision path, where word tokens attend to image grid cells, and a vision-guides-language path, where grid cells attend back to words. It then regresses a box without proposals. We stack $L$ such layers and attach a lightweight anchor-free box head after each one.
At layer $l$, let $A_l \in \mathbb{R}^{W \times G}$ be the word-to-grid attention, with $W$ words and $G$ grid cells. Let $B_l \in \mathbb{R}^{G \times W}$ be the grid-to-word attention, averaged over heads and row-normalised. Let $m_l \in \{0,1\}^G$ mark the grid cells inside the predicted box $b_l$, and let $\pi$ be word weights that emphasise content words, taken from the model's own pooling. We define the exit score as
$$ g_l = \underbrace{\pi^\top A_l\, m_l}_{\text{text looks inside the box}} \cdot \underbrace{\frac{\pi^\top \operatorname{diag}(A_l B_l)}{\pi^\top \mathbf{1}}}_{\text{round trip returns to the same word}}, $$
and exit at the first layer where $g_l \ge \tau$. The first factor is the fraction of the sentence's visual attention that lands inside the box the head is proposing. The second is a cycle-consistency term. A word's attention goes out to the image, the attended cells attend back to the words, and we measure how much mass returns to the original word. Text-prior shortcuts should fail this test: the box sits where the sentence alone suggests, but the grid cells there are not actually attending to the describing words. The threshold $\tau$ is set on validation data to meet a target average compute, following the budgeted calibration used in DeeR-VLA.
The score costs almost nothing, because the attention matrices are already computed. The attention is parameter-free in the sense of CALIP, which showed that raw symmetric attention in an aligned space carries usable signal. We borrow that observation but use the attention as a verifier rather than as a feature enhancer.
Research Program
The program has three stages.
Build the multi-exit grounding model. We train intermediate box heads in two ways and compare them, since joint training of all heads is known to compete with deep-layer capacity. The first is joint training with a weighted multi-exit loss plus self-distillation, in which intermediate boxes mimic the final head, as in SCAN. The second is decoupled training in the FastBERT style: the backbone is frozen and only the heads are trained. The observation in CMSA that middle layers attend to attributes and spatial words while deep layers attend to category nouns predicts a specific pattern. Intermediate heads should do well on short category-dominated expressions and poorly on relational ones. We will check this directly.
Study when agreement and confidence diverge. We will measure how often each rule fires on instances that turn out to be wrong, and on which expression types this happens.
A contingent extension. If the text stream's attention stabilises well before the vision stream's, we test an asymmetric variant. The word tokens' keys and values are cached and frozen, while grid cells continue to attend to them. This keeps the feedback loop that DyVTE and HiDrop cut off, but at lower cost. We treat this as secondary. It is pursued only if the main exit result holds.
Baselines
The naive reference is the full-depth co-attention model, run at batch size 1 on the same GPU. Every accuracy-versus-compute curve is anchored to it. Published full-depth numbers give scale:
- PFOS reports 66.90% accuracy on the ReferItGame test set at roughly 20 ms per image.
- MUTATT reports 78.35% on RefCOCO in the detection setting and 69.03% on RefCOCOg.
- CMSA reports 58.32% IoU on the UNC validation set for segmentation.
- CEFNet reports 51.93% overall IoU on G-Ref at 50 FPS.
We will re-train our own full-depth model rather than compare against these numbers across codebases. I found no reported result on Lacuna for any early-exit method on referring expression comprehension or segmentation. Every adaptive baseline below is therefore run by us.
The second baseline is a static shallow model: the same architecture truncated to $k$ layers and trained alone, with $k$ chosen to match the adaptive methods' average compute. It asks whether adaptivity matters at all.
The third is calibrated confidence exit. We take the head's objectness or box score, apply per-exit temperature scaling as in The Right Tool for the Job, and set per-exit thresholds with the heuristic from Multimodal Adaptive Inference.
The fourth, and the strongest competitor, is stability-based exit. This is patience in the PABEE sense adapted to boxes: exit once consecutive heads' boxes overlap above an IoU threshold for $t$ layers. We also run the $L_2$ box-coordinate version of the DeeR-VLA consistency criterion. Stability is strongest because it already filters out one-layer confident flukes and needs no access to internals.
The fifth is a learned gate in the style of DyVTE: a small MLP on pooled hidden states predicts whether to exit, trained on the same data.
Two ablations isolate the key ingredient. The first keeps only the text-to-grid factor of $g_l$, removing the round trip, which tests whether bidirectionality in the signal matters. The second trains a unidirectional backbone, where words attend to the image but not the reverse, with the same exits. This tests whether the backbone's bidirectionality matters.
All methods share the trained backbone and heads where applicable, the training data, and the validation split used for threshold search. Each gets the same search budget: an equal number of threshold or patience settings. "Beating" a baseline means higher accuracy at matched mean FLOPs across the budget range, summarised as the area under the accuracy-compute curve, with at least three seeds. It also means a lower error rate among early-exited instances.
Why not just…?
Why not just compress the model statically? Static distillation or pruning gives every input the same reduced depth. On text, FastBERT and BERT Loses Patience report that adaptive exits dominate static compression, with 61% of inputs in one FastBERT configuration exiting after the first layer. That this carries over to grounding is not established. The matched-compute truncated model settles it. If it matches the adaptive curves, grounding difficulty is too uniform for exits to pay off, and we would report that.
Why not just prune background grid cells? The main cost in these models is attention over grid cells, so pruning them, as FastV-style methods do in LLMs, attacks cost directly. DyVTE reports that depth-wise exit and within-layer token pruning are complementary, reaching a 51.5% compute reduction with a 0.7% average drop when combined. So we treat pruning as orthogonal and include a grid-pruning plus agreement-exit combination. Pruning alone is also a baseline at matched compute.
Why not just stop when consecutive boxes stop moving? This is the real threat. Box stability is simple, and it may catch most of what agreement catches. Our bet is that stability and agreement fail on different instances. A text-prior box can be stable across layers precisely because it ignores the image. Agreement tests the image directly. The diagnostic experiments below target exactly this case. If stability matches agreement on those tests as well as on overall curves, the conclusion is that attention-derived exits are unnecessary for grounding, and the paper would say so.
Why not just add early exits to a unidirectional model? This would be cheaper per layer. But PFOS reports that combining both attention directions gains nearly 10% over concatenation on ReferItGame, and MUTATT reports gains from mutual guidance. The unidirectional-backbone ablation measures whether those gains survive once both models can exit early.
Proposed Experiments
We train on RefCOCO, RefCOCO+, and RefCOCOg for box grounding, and on ReferItGame for transfer. Where the segmentation extension is run, following CMSA and CEFNet, the heads predict masks, and the same agreement score uses mask cells in place of box cells. The metrics are:
- accuracy at IoU 0.5 (or mean IoU for masks) against mean FLOPs per instance;
- wall-clock latency at batch size 1, the regime where The Right Tool for the Job notes exits actually speed things up;
- the premature-error rate: the fraction of early exits that are wrong but that the full model gets right.
Results are stratified by split and expression type. RefCOCO testA and testB separate people from other objects. RefCOCO+ forbids location words, which removes one route to text-prior shortcuts. RefCOCOg has long relational expressions.
The decisive checks are two stress tests that target text-prior confidence.
Mismatched pairs. Each expression is paired with a random image from the same category split. The correct behaviour is to not exit early; ideally the full model also signals low agreement. We compare how often each rule exits early on these pairs.
Referent-removed images. The referent region is inpainted out of the image. Any confident early box is then a text-prior guess.
The proposal succeeds if agreement exits fire substantially less often than confidence and stability exits on both tests. It must also match or beat them on the clean accuracy-compute curves. It fails if stability is equally robust. Finally, we test the prediction from CMSA: relational RefCOCOg expressions should exit later than short RefCOCO ones under agreement, and the gap should be smaller under confidence exits. This would show that agreement tracks linguistic difficulty rather than head calibration.
References
- The Right Tool for the Job: Matching Model and Instance Complexities
- BERT Loses Patience: Fast and Robust Inference with Early Exit
- FastBERT: a Self-distilling BERT with Adaptive Inference Time
- Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exiting
- DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
- HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
- Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention
- CALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention
- SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models
- Cross-Modal Self-Attention Network for Referring Image Segmentation
- Mutatt: Visual-Textual Mutual Guidance For Referring Expression Comprehension
- Encoder Fusion Network With Co-Attention Embedding for Referring Image Segmentation