Divide-and-Conquer Grounding Across a Visual Feature Pyramid

When a model segments "the small wooden chair on the far left next to the sofa," today's best parameter-efficient systems compress the whole sentence into one language vector and inject that same vector at every depth of a frozen vision network. That network describes the image at several scales. Early layers see edges, textures and colors at high resolution. Deep layers see object categories at low resolution. A single sentence vector cannot tell each scale which part of the sentence it should check. This proposal splits the referring expression into short sub-phrases. Each sub-phrase is matched independently against the feature scale whose granularity fits it: object category against deep features, color and material against shallow features. The per-phrase relevance maps are then merged with a learned, content-dependent gate. The sentence-processing side borrows divide-and-conquer prompting from large language models, and the image side borrows multi-scale feature fusion from dense vision.

The bet is that independence is the useful ingredient. Divide-and-conquer prompting works for language models because each sub-problem is solved in isolation, which stops errors from one part spreading into the others. We expect the same to hold when a compositional expression is matched against an image. The claim is falsifiable and narrow. Gains should concentrate on long, multi-attribute expressions, and a strong learned alternative (letting every scale attend over all words) should fail to reproduce them. If that alternative matches us, the explicit decomposition is unnecessary and we will say so.

Core Claim

Decompose a referring expression into sub-phrases. Solve each sub-phrase against a scale-matched feature level without seeing the other sub-phrases. Merge the resulting maps with an attentional gate. On compositional expressions, this beats injecting a single global prompt at all scales, and it also beats learned per-scale attention over the full sentence, at equal trainable-parameter budget on the same frozen backbone.

What Prior Work Does and Leaves Open

The closest system is WIMFRIS. It adapts a frozen DINOv2-Large backbone to referring image segmentation, the task of producing a pixel mask for the object a sentence describes. It adds a hierarchical fusion "neck" built from Mamba, a sequence model that processes tokens by a running scan. To stop that scan from forgetting distant tokens, WIMFRIS splits the image into windows and prepends one global language token to every window. This paper supplies our backbone, training budget and strongest baseline. It also motivates the proposal in two ways. Its learnable per-stage emphasis weights show that early layers need more adaptation than deep ones, yet the language signal it gives each stage is identical. It also reports failures on objects that cross window boundaries and on referents described relative to distant anchors. We keep its backbone and its parameter budget. We replace the single prepended token with scale-routed sub-phrase maps.

Two other papers bring prompts and feature pyramids together, but neither structures the prompt itself:

  • AuralSAM2 fuses a frozen SAM2 feature pyramid with audio and text through cross-attention to produce sparse and dense prompts for the mask decoder. It needs an extra contrastive loss so that dense visual signals do not drown out the sparse audio and text signals.
  • Referring Segmentation via Encoder-Fused Cross-Modal Attention Network uses language-guided multi-scale dynamic filtering and reports about 2.3 points of J&F from it.

In both, the whole expression conditions every scale uniformly. We differ by making the language side hierarchical and the matches independent.

APL supplies part of the method. It shows that splitting an anchor's description into spatial, color and category prompts helps weakly supervised grounding, reporting +6.44% on RefCOCO over RefCLIP. But it works on a flat, single-scale anchor grid and maps colors to words with a lookup table. We carry its attribute split onto pyramid levels, using continuous matching instead of a lookup.

On the prompting side, two papers formalize divide-and-conquer prompting: When Splitting Makes Stronger and Prompting LLMs with Divide-and-Conquer Program. The method has three steps: decompose the task, solve sub-tasks independently, merge the results. They report an F1 of 74.84 versus 46.85 for chain-of-thought on HaluEval-Summary. They also supply a constraint we take seriously: decomposition does not help when sub-problems are tightly coupled. A relational phrase like "next to the sofa" is coupled to its anchor. We therefore treat relations as a second, dependent level rather than pretending they are independent.

The hierarchical prompt methods each build coarse-to-fine prompts, but none ties a parsed language hierarchy to the internal scales of a vision-language grounding model:

  • MSGCOT chains coarse-to-fine prompts over graph scales.
  • HCVP splits visual prompts into domain-level and task-level parts.
  • IVPT aggregates part concepts into macro-concepts across ViT layers.

HMAW is a multi-agent prompting workflow whose hierarchy adds 200–700% inference latency. That overhead is a cost our design avoids by decomposing once, cheaply, outside the image model.

For merging, Attentional Feature Fusion supplies the gating method. It shows that plain addition of feature maps from different scales computes its fusion weights from inputs that were never aligned, which the authors call an "initial integration bottleneck." Its fix is a gate computed from parallel local and global channel-attention branches.

Research Program

Step 1: decompose each expression. A dependency parser, or a small language model run once offline, splits each expression into three kinds of sub-phrase. Head phrases name the category ("chair"). Attribute phrases describe appearance ("small," "wooden"). Relational phrases have the form relation plus anchor ("next to the sofa," "far left"). Decomposition happens once per expression, with no repeated calls to a large model, so it avoids HMAW-style latency. Parse quality is measured on a hand-checked sample of 500 G-Ref expressions.

Step 2: solve sub-phrases in parallel. The frozen DINOv2-Large backbone runs once and produces a four-level pyramid. Each sub-phrase embedding is matched against the pyramid by a lightweight scorer. The scorer is a small convolution plus a dot-product with the phrase, applied per level. It yields a relevance map $S_{k,\ell}$ for sub-phrase $k$ at level $\ell$. Each sub-phrase sees only itself, never the other sub-phrases.

Step 3: route each sub-phrase to scales. Each sub-phrase's maps are weighted across levels with a soft routing vector. We compare three ways of setting it:

  • a fixed prior: heads to deep levels, attributes to shallow levels;
  • a routing vector learned from the phrase embedding;
  • uniform routing, as an ablation.

Step 4: handle relations recursively. This is the one place we break independence on purpose. The anchor phrase is first solved as its own head-plus-attribute sub-problem. The relation scorer then sees the anchor's map together with the deep features. Because deep levels are matched globally rather than in windows, this step also targets WIMFRIS's failure on referents far from their anchors.

Step 5: merge with a gate. Routed maps are upsampled to the finest resolution and combined as

$$M = \sum_{k} g_k \odot \tilde S_k,$$

where:

  • $\tilde S_k$ is sub-phrase $k$'s routed, upsampled map;
  • $g_k$ is a per-pixel gate computed from $\tilde S_k$ and the shallow visual features by local and global channel-attention branches, following Attentional Feature Fusion;
  • $\odot$ is element-wise multiplication.

The merged map $M$ conditions the same mask decoder WIMFRIS uses. All trainable modules together are held to WIMFRIS's 1.3–2.2% of backbone parameters.

Baselines

The naive reference point is the frozen DINOv2-Large backbone with a single pooled sentence embedding injected identically at every pyramid level and an additive merge. This is roughly a parameter-efficient adapter with no fusion neck. It sets the floor and shows how much any language-aware multi-scale design adds. It is trained with the same data, schedule and parameter budget as every other variant.

The strongest existing method is WIMFRIS. It reports an average of 73.4 mIoU across the standard RefCOCO, RefCOCO+ and G-Ref splits with DINOv2-Large, and 78.2 mIoU when trained on the mixed datasets, while tuning 1.3–2.2% of backbone parameters. The file does not give per-split numbers, so we will reproduce WIMFRIS from its description with our training pipeline. We report the reproduced and published averages side by side and compare against the reproduction for per-split and per-bucket analysis. Beating it means exceeding the reproduced average at matched trainable parameters and matched GPU-hours. It also requires a larger margin on the compositional bucket than on short expressions.

We considered Referring Segmentation via Encoder-Fused Cross-Modal Attention Network as a second competitor, since its language-guided multi-scale filtering is the nearest "whole sentence at every scale" design. However, no reported result was found on Lacuna for it on RefCOCO, RefCOCO+ or G-Ref. We therefore treat its filtering module as a component to re-implement inside our matched pipeline, not as a cited number.

The threat baseline has every pyramid level run its own cross-attention over all word tokens, with level-specific queries. It is sized to the same parameter count as our method. This baseline could learn scale-specific word selection without any explicit parsing. If it matches us, the decomposition is not doing the work.

Ablations isolate the key ingredients, one at a time:

  • Independence: sub-phrases are concatenated and matched jointly at each level.
  • Scale matching: uniform routing.
  • Learned versus fixed routing.
  • Gate versus additive merge.
  • Recursion: relations are treated as independent sub-phrases.

Each ablation uses the same training budget and three random seeds. We count a difference only when it exceeds the seed spread.

Why not just…?

Why not just let each scale attend over all the words? This is the real threat. Per-level cross-attention can in principle learn to pick color words at shallow levels and nouns at deep ones. Nothing in the file shows that it fails, though AuralSAM2 needed an auxiliary contrastive loss to keep cross-attention from being dominated by visual features. Our argument for decomposition is about independence rather than selection. Attention mixes all words into each query, which is the sequential entanglement that divide-and-conquer prompting removed in language models. The parameter-matched threat baseline settles this directly. Our hypothesis predicts that it matches us on short RefCOCO expressions and trails on expressions with three or more parsed sub-phrases. If it does not trail, we drop the claim.

Why not just fine-tune the whole backbone? Full fine-tuning changes the comparison from "how to inject language" to "how much capacity is available." WIMFRIS already reaches 73.4 mIoU with about 2% of parameters tuned. HCVP reports prompt modulation beating full-fine-tuning ERM out of distribution (81.30% vs 75.13% on the PACS Sketch domain). We will run one full fine-tuning reference on G-Ref to bound the gap, but the main question is posed at a fixed parameter budget.

Why not just rewrite the expression with a language model and feed the cleaner sentence to WIMFRIS? Rewriting, for example with chain-of-thought canonicalization, still ends in one vector shared by all scales. So it cannot test whether scale routing matters. It also costs a large-model call per query, the overhead HMAW shows to be 200–700%. We include this as a baseline: WIMFRIS reproduced, fed parser-normalized sentences. That separates "better text" from "better routing."

Why not just merge the maps by addition, as a feature pyramid network does? Attentional Feature Fusion reports that a gated fusion lets ResNet-50 reach 20.2% ImageNet error, versus 23.2% for a plain ResNet-101. That is evidence about channel fusion inside a backbone, not about merging phrase maps. The gate-versus-add ablation tests whether the effect carries over. If addition matches the gate, we keep addition and simplify the method.

Proposed Experiments

Data and metrics. We train on RefCOCO, RefCOCO+ and G-Ref (both splits) following WIMFRIS's protocol, plus a mixed-data run for comparison with its 78.2 mIoU figure. We report mIoU and overall IoU per split.

Expression buckets. The headline analysis stratifies the test sets by parsed sub-phrase count (one, two, three or more) and by the presence of relational phrases. RefCOCO+ bans absolute spatial words, so it isolates the attribute-to-shallow-level route. G-Ref's long expressions stress decomposition. We also bucket by object size relative to WIMFRIS's window size, to test whether global deep-level matching reduces fragmentation on large objects.

Decisive checks. There are three:

  • the gap between our method and the parameter-matched per-level attention baseline should grow with sub-phrase count;
  • removing independence should erase most of that gap;
  • the learned routing vectors should, without supervision, place attribute phrases on shallower levels than head phrases. We measure this by the average routed level per phrase type.

Robustness and cost. For robustness to parse errors, we inject controlled parse corruptions (dropped or merged sub-phrases) and plot accuracy against corruption rate. We report throughput against the WIMFRIS reproduction, expecting a modest overhead because the backbone runs once and only the light per-phrase scorers repeat.

What would kill the idea. The idea fails if the per-level cross-attention baseline matches us within seed noise on the three-or-more bucket. It also fails if the learned routing stays uniform. Either outcome would itself be a useful result. It would show that, for frozen-backbone grounding, the scale-structure argument behind hierarchical prompting does not survive a fair learned competitor.