Referring image segmentation means drawing the pixel mask of the one object a sentence describes, such as "the man holding the umbrella" or "the dog on the right". Current models find the right object reliably, but nobody has checked whether they trace its outline well. They are scored with mask IoU (the overlap between predicted and true masks), and object interiors dominate that score. Separately, the encoder-decoder literature shows that the high-resolution "skip connections" which carry fine detail from early layers to the decoder also carry noise. These early feature maps contain every edge in the image, including the edges of the neighboring person, the background clutter, and the second dog. This note proposes a skip connection gated by the words of the expression, together with an evaluation built to tell whether language actually helps at the boundary. The gate is computed per pixel from attention between each early-layer pixel and each word, so the decoder receives high-resolution detail only where the sentence says the referent is.
The claim is narrow and testable. In referring segmentation, outline errors come mostly from choosing the wrong edges, not from lacking edges. The edges are present in the high-resolution features, but the decoder cannot tell which ones belong to the described object where it touches another object of the same kind. If this is right, language-gated skips should improve boundary-sensitive metrics far more than mask IoU. The gains should cluster at boundaries where the referent touches a same-category instance. They should also disappear when the gate alone is given the wrong sentence. If gating driven by the decoder's own (already language-aware) state matches the word-driven gate, the claim is false, and the experiments are built to show that.
What exists and where this sits
Three referring-segmentation systems define the landscape, and none treats the skip path as the place where language should act. LAVT inserts pixel-word attention into every stage of a Swin Transformer encoder, with vision as queries and words as keys and values, plus a learned gate. It then decodes with a plain top-down fusion of the stages. Its skip features are therefore already somewhat language-aware, which is the most important fact a reviewer will raise. Our design must beat "already language-aware features passed through an ordinary decoder," not a language-blind baseline. EF-Net makes vision and language update each other inside the encoder. It keeps cost manageable by computing pixel-to-word affinities of size $HW \times T$ rather than pixel-to-pixel ones, a trick we reuse. It works on video and never evaluates boundaries. VATEX and its journal version Vision-Aware Text Features refine text features with visual context inside a query-based decoder. They leave fine detail to mask attention and report only region overlap.
The motivation comes from the encoder-decoder side. UCTransNet found that on the GlaS gland dataset a U-Net without skip connections beats one with them. It fixed this with channel-wise transformers that let decoder features attend over multi-scale encoder channels, raising Dice from 85.45% to 90.18%. That work is unimodal, and its decoder-queried attention is exactly the strongest competitor to our word-queried gate. Cross-Modal Deep Face Normals and LaDI-VTON gate skip paths with hard switches or binary masks. This supports the idea that skip traffic should be controlled, while showing that nobody has controlled it with a continuous signal from language.
The evaluation gap comes from Boundary IoU, which shows that Mask R-CNN on large COCO objects reaches 66.0 mask AP but only 25.9 boundary AP. It also shows that PointRend raises mask IoU by 8% but boundary IoU by 22%. A real boundary improvement can therefore hide inside a small mask-IoU change. gen2seg adds a warning that constrains our baselines: SAM scores 16.8 mIoU on the thin-structure iShape benchmark against 51.4 for a generative backbone, so "just refine with SAM" is not a safe fix for fine detail. PolyFormer sidesteps dense decoding by emitting polygon vertices. It is a strong overall competitor but answers a different question.
Core mechanism
At skip stage $s$, let $F_s \in \mathbb{R}^{H_sW_s \times C}$ be the encoder features and $E \in \mathbb{R}^{T \times C}$ the projected word embeddings of the expression. Let $U_s$ be the upsampled decoder state arriving from the coarser stage. Each pixel first gathers a word context $c = \mathrm{softmax}(F_s E^\top/\sqrt{C})\,E$, at cost $O(H_sW_sT)$ as in EF-Net, rather than quadratic in pixels. A gate is then computed per pixel and per channel group, and only the gated skip is fused:
$$ g_s = \sigma\big(\mathrm{MLP}([F_s,\; c,\; U_s])\big), \qquad \tilde F_s = g_s \odot F_s , $$
where $\sigma$ is the logistic function, $[\cdot]$ denotes concatenation, and $\odot$ is element-wise multiplication. The gate is initialized near one, so training starts from LAVT's ordinary decoder. The gate sees the decoder state $U_s$, so the comparison with decoder-only gating is a clean removal of the word term $c$ rather than a change of architecture. We add no extra supervision for the gate. The only loss is LAVT's mask loss, so any boundary gain comes from routing rather than from a new objective.
Research program
We reproduce LAVT with the Swin-B backbone and BERT text encoder on RefCOCO, RefCOCO+ and G-Ref. We confirm the reproduction against the published overall IoU before changing anything. We then replace the decoder's skip fusion with the gated version at the two highest-resolution stages, where edges live, and separately at all stages.
The analysis has three parts. First, we report Boundary IoU alongside overall and mean IoU on all three benchmarks. As far as the file shows, no referring-segmentation paper reports it, so this alone is a contribution. Second, we use COCO instance annotations to split each referent's boundary pixels into three groups: pixels touching another instance of the same category, pixels touching an instance of a different category, and pixels touching background. We report boundary error per group. Our hypothesis predicts gains concentrated in the same-category group. Third, we build an evaluation-only subset of roughly 500 G-Ref expressions whose referents have low area-to-perimeter ratio, such as people with extended limbs, held objects, animals with leashes, and bicycles. We re-annotate these masks finely, because COCO polygons are themselves coarse and would cap any boundary metric. We will report how much of the measured boundary error on the original labels is annotation noise, using agreement between original and re-annotated masks.
Baselines
The naive reference point is LAVT exactly as published and reproduced, with its plain top-down decoder. The cited number is 72.73% overall IoU on RefCOCO, 62.14% on RefCOCO+, and about 61% on G-Ref, from LAVT. No reported Boundary IoU result was found on Lacuna for LAVT or any referring-segmentation model, so we compute it ourselves for every system. This baseline matters because its skip features already contain language through the encoder's pixel-word attention. Beating it shows that where language enters matters, not merely whether it enters.
The strongest competitor on the boundary question is decoder-queried skip attention in the style of UCTransNet's channel cross-attention, transplanted into LAVT. We implement it as our gate with the word term $c$ removed, keeping parameter count matched by widening the MLP. The decoder state in LAVT is already conditioned on the sentence, so this baseline captures the objection that the coarse mask already tells you which object it is. We also run the full channel transformer from UCTransNet as an unconditioned skip filter. For overall accuracy we compare against PolyFormer, which reports a 5.40% absolute gain over LAVT on RefCOCO+. The file gives no absolute value, so we cite that gain and rerun the released model to compute Boundary IoU. We include VATEX where its released model allows boundary evaluation.
All LAVT-derived variants share the same backbone, data, schedule, resolution and seed set. Each gets the same small learning-rate sweep, and results are averaged over three seeds. Beating the decoder-queried baseline means a Boundary IoU gain larger than the seed spread, concentrated at same-category contact boundaries, with overall IoU no worse than LAVT. A gain only in overall IoU, or a boundary gain shared equally by the decoder-queried gate, does not count.
The ablation that isolates the key ingredient is a mismatched-gate control. At test time the encoder and decoder receive the correct expression, but the gate's word tokens come from a different expression for a different object in the same image. If the gate really uses language to select edges, boundary quality should fall, mostly at contacts with the object the wrong sentence describes, while interior overlap barely moves.
Why not just…?
Why not just add a boundary loss or a higher-resolution decoder to LAVT? This addresses edges that are missing, while our hypothesis concerns edges that are present but wrongly selected. PointRend's large boundary gains in Boundary IoU show that point-wise high-resolution refinement is a real lever. We therefore include a LAVT variant with a boundary-weighted loss and a variant with an extra full-resolution decoder stage at matched compute. If either matches the gate at same-category contacts, the selection hypothesis is wrong. If they improve background-contact boundaries but not same-category ones, the two fixes are complementary.
Why not just prompt SAM with LAVT's box or mask? gen2seg reports SAM at 16.8 mIoU on iShape's thin structures, so SAM refinement is a poor bet exactly where outlines are hard. SAM also never sees the sentence, so it cannot decide whether the held umbrella belongs to "the man holding the umbrella." We still run LAVT-mask-prompted SAM on the fine-annotated subset because it is cheap. A win there would be a sobering result.
Why not just let the decoder state choose, as UCTransNet's cross-attention does? This is the real threat. LAVT's decoder already knows roughly which object is meant, and on most boundaries that may be enough. The proposal does not argue this away. The decoder-only gate is a matched ablation, the per-group boundary analysis shows where any difference lives, and the mismatched-gate control shows whether the words are used at all. If the two gates tie everywhere, we will report that language is not needed past the encoder. That is a useful negative finding for anyone designing these decoders, and the Boundary IoU audit of existing models would stand on its own.
Decisive checks
The idea survives only if three results hold together. Word-queried gating must beat decoder-queried gating on Boundary IoU on G-Ref and on the fine-annotated subset, by more than the seed spread. The gain must be largest at boundaries shared with a same-category instance. Feeding the gate a mismatched sentence must erase the gain while leaving interior overlap nearly unchanged. Overall IoU should stay within noise of LAVT's 72.73% on RefCOCO. A large overall-IoU jump without a boundary-specific pattern would suggest the gate acts as extra capacity, not as language-directed edge selection.
References
- LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
- Referring Segmentation via Encoder-Fused Cross-Modal Attention Network
- Vision-Aware Text Features in Referring Image Segmentation: From Object Understanding to Context Understanding
- UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-Wise Perspective with Transformer
- Cross-Modal Deep Face Normals With Deactivable Skip Connections
- LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On
- Boundary IoU: Improving Object-Centric Image Segmentation Evaluation
- gen2seg: Generative Models Enable Generalizable Instance Segmentation
- PolyFormer: Referring Image Segmentation as Sequential Polygon Generation