Distance-Calibrated Text-to-Pixel Similarity for Referring Segmentation

Referring image segmentation asks a model to cut out the object a sentence describes ("the thin branch above the bench"). Current systems such as CRIS score every pixel by how similar its feature vector is to the sentence's feature vector, then train that score with a yes/no target: a pixel is either on the object or off it. A pixel one step outside the boundary and a pixel in the far corner of the sky receive the same push, and nothing pins down what value the similarity should take at the boundary. We propose to change the target, not the architecture. Each pixel's text-to-pixel similarity would be trained to reproduce a smooth geometric map: high along the object's skeleton (its medial line), exactly a fixed value at its boundary, and falling with distance outside. In other words, the cross-modal similarity itself becomes a distance transform, a map that records for every pixel how far it is from a target shape.

The reason this matters goes beyond sharper edges. Work on the modality gap shows that image and text embeddings sit in separate regions of the shared space, offset from one another, and that the size of this offset changes downstream behavior (Mind the Gap). In dense grounding the same offset shows up locally. How similar the referent's pixels are to the sentence, compared with how similar the background is, plausibly drifts from one expression to the next, so a single global mask threshold is wrong for many expressions. Training similarity to equal a fixed function of signed geometric distance gives that offset a fixed meaning, so that the boundary lies at a known similarity value for every sentence. The claim is testable, and it is the part of this proposal that a plain "add a distance loss" paper would not test.

Core Claim

Write $s(p) = \cos(t, v_p)$ for the cosine similarity between the sentence embedding $t$ and the decoder's embedding $v_p$ at pixel $p$. Write $y(p)$ for a signed geometric target. Inside the referred mask, $y$ is the skeleton-aware energy $E(p) = (d_b/(d_s+d_b))^{\alpha}$ from Structure-Preserving Instance Segmentation via Skeleton-Aware Distance Transform. Here $d_b$ is the distance to the boundary, $d_s$ the distance to the skeleton, and $\alpha \approx 0.8$. The energy equals 1 on the skeleton and 0 on the boundary. Outside the mask, $y(p) = -\min(d_o(p)/\tau, 1)$, where $d_o$ is the distance to the mask and $\tau$ is a band width in pixels. We train

$$\mathcal{L} = \frac{1}{|P|}\sum_{p \in P} w(p)\,\big(s(p) - (c_0 + c_1\, y(p))\big)^2,$$

where $c_0$ is a fixed cosine value that defines the boundary, $c_1 > 0$ sets the dynamic range, and $w(p)$ balances inside and outside pixels. At test time the mask is $\{p : s(p) > c_0\}$, with the same threshold for every expression.

The inside target has two useful properties. Because $d_b/(d_s+d_b)$ is a ratio, it does not depend on object size, which matters on RefCOCO where referents range from a cup to a whole bus. Because SDT builds it to have a single ridge, it discourages the split, multi-peaked responses that plain Euclidean distance maps give on objects with narrow necks.

We make three falsifiable claims. First, compared with the binary text-to-pixel contrastive loss, the best threshold for each expression will vary much less, and the gap between using that per-expression oracle threshold and using the fixed threshold will shrink. Second, boundary quality and connectivity will improve most on thin and elongated referents. Third, both effects depend on putting the distance target into the cross-modal similarity. A separate distance head that sits beside an unchanged binary similarity should not reproduce them.

Research Program

We build on CRIS (CRIS: CLIP-Driven Referring Image Segmentation), which supplies the method we modify. It feeds sentence features into pixel tokens through a cross-attention decoder and trains the result with a text-to-pixel contrastive loss that pulls foreground pixels toward the sentence and pushes background pixels away. We keep its encoder, decoder and training schedule, and replace only that loss with $\mathcal{L}$. All targets are computed once, offline, from the ground-truth masks. The method therefore adds no network module, which avoids the learned distance-approximation network of Deep Distance Map Regression Network with Shape-aware Loss. It also adds no second forward pass, unlike the alignment filtering in AMLRIS, which reports a 17.2% per-epoch training overhead.

The motivation comes from unimodal distance regression. Multiscale Centerline Detection by Learning a Scale-Space Distance Transform shows that classification score maps plateau on thin structures and that regressing a decaying distance creates ridges that are easy to extract. DEF reports that regressing a distance-to-feature field, rather than classifying features, raised recall at 2 mm on real scans from 29.5% to 91.7%. Neither paper involves language. Our question is whether the same benefit carries over when the scalar field is a similarity between two modalities.

Two retrieval papers constrain the design and support it from the contrastive side. Soft Contrastive Learning for Visual Localization and Data-Efficient Large Scale Place Recognition with Graded Similarity Supervision replace binary pairs with continuous scores derived from geometric distance. The second reports that this removes the need for hard-negative mining. Both work within a single modality and score whole images rather than pixel grids. We carry their graded supervision down to the pixel level and across modalities. Descriptive Image-Text Matching with Graded Contextual Similarity grades image-text matches by how specific a caption is, but only for global retrieval. It also reports a small loss in within-modality retrieval, which warns us to check that pixel embeddings stay discriminative. We do this with linear probes for object category.

The analysis has three parts. We measure the local modality gap: for each expression, the difference between the mean similarity of referent pixels and the sentence embedding's similarity to its nearest pixel cluster, in the spirit of Mind the Gap's centroid-offset measure. That paper studies only global centroids of each modality; we track the per-expression version of this offset, and how much it varies across expressions, during training. We measure how the oracle threshold is distributed across expressions. And we look for ordering: whether similarity falls monotonically with distance outside the object, compared with binary training. The main risk we are watching for is that, as the inside target falls toward the boundary, the model loses its sharp separation between the referent and a nearby same-category distractor. To catch this we evaluate separately on expressions with a same-category distractor in the image, using COCO category labels.

Baselines

The naive reference point is CRIS trained exactly as published, with its binary text-to-pixel contrastive loss and a fixed threshold. It reports IoU of 70.47% on RefCOCO, 62.27% on RefCOCO+ and 59.87% on G-Ref (CRIS). We retrain it ourselves with identical seeds, schedule and hardware, so that every comparison is against our own reproduction. The published numbers serve only to check that the reproduction is sound.

The strongest existing methods in the file are AMLRIS (AMLRIS) and MaskRIS (MaskRIS). AMLRIS removes poorly aligned pixels with a hard filtering mask and reports state-of-the-art results on all eight RefCOCO-family splits. MaskRIS adds paired image and word masking with self-distillation and reports oIoU gains of 1.82 points on RefCOCO and 2.76 points on RefCOCO+ over CARIS. The file gives AMLRIS only as relative gains (up to 1.92% mIoU on RefCOCO+ and an average of 3.50% across seven perturbation scenarios), with no absolute IoU. It gives no absolute MaskRIS number on RefCOCOg. We therefore apply both training strategies to our CRIS reproduction, using equal epochs and an equal hyperparameter search of 12 configurations each. Both act on inputs or pixel selection, while ours acts on the target, so we also test each in combination with our loss. Beating them means a higher mean IoU and higher boundary F-score on the full splits, averaged over three seeds with non-overlapping confidence intervals. On the thin-object subset we expect a larger margin.

We also run three ablations that isolate the key ingredient. The first uses the same distance maps only as per-pixel weights in the binary loss. The second adds a separate distance-regression head while keeping binary text-to-pixel similarity. The third replaces the skeleton-aware inside target with a plain truncated Euclidean distance. For thin objects, the file reports no result on Lacuna for boundary F-score, connected-component error or the thin-object subset in referring segmentation, so these comparisons rest entirely on our own runs.

Why not just…?

Why not just weight the binary loss by distance to the boundary? This is the most serious threat, because it uses the same geometric information at almost no cost. The difference is that weighting changes how hard each pixel is pushed, but not where its similarity should end up. The similarity can still plateau, and its offset can still drift from one expression to the next. Our first claim predicts that weighted binary training improves boundary F-score but leaves the oracle-threshold gap mostly intact. If it closes that gap as well as our loss does, then the cross-modal framing adds nothing, and we will report the weighted loss as the recommended method.

Why not just add a distance-map head next to the existing decoder? This is how Deep Distance Map Regression Network uses distance maps in medical segmentation, where it reaches 0.7463 Dice on LiTS. It would probably sharpen masks in referring segmentation too. But the extra head can learn the geometry while the text-pixel similarity that decides the mask stays binary in character. The second ablation tests this directly. Our hypothesis fails if the extra head matches our loss on threshold variance and on the distractor subset.

Why not just tune the threshold or shift the embeddings, as Mind the Gap does? Mind the Gap raised CIFAR-10 zero-shot accuracy from 0.9013 to 0.9081 by moving embeddings along the offset between modalities, but that was a single global shift. If the offset varies by expression, one shift cannot fix it. We test this by fitting the best global threshold on a validation split, and also by training a small head that predicts a threshold for each expression. The per-expression head is a real competitor. If it recovers most of the oracle gap, the remaining value of our loss lies only in connectivity and boundary shape, and we will say so.

Why not just clean up masks with DenseCRF or morphological operations? CRIS notes that earlier systems relied on DenseCRF, which is expensive and cannot be trained end to end. Post-processing also cannot reconnect a fragmented response when the underlying score has no ridge to follow, which is the argument made in Structure-Preserving Instance Segmentation via Skeleton-Aware Distance Transform. We apply DenseCRF to the reproduced CRIS outputs as an extra reference point and compare connected-component error on thin objects.

Proposed Experiments

We train and evaluate on RefCOCO, RefCOCO+ and RefCOCOg, using all standard splits. RefCOCO+ bans absolute location words, so it tests whether the geometric target helps when the sentence relies on appearance rather than position. The metrics are mean IoU and overall IoU, boundary F-score at several pixel tolerances, and connected-component error (the number of predicted components minus the number in the ground truth). We also report the variance of the per-expression oracle threshold and the oracle-minus-fixed IoU gap. We construct a thin-object subset by ranking referents by skeleton length divided by mask area and keeping the top decile. We construct a distractor subset from images that contain another instance of the referent's category.

The decisive check is a two-by-two comparison. One factor is binary versus distance target; the other is whether the distance enters the similarity itself or a separate head. If the gains in threshold stability and thin-object connectivity appear only when the distance is in the similarity, the cross-modal claim holds. If they appear equally with the separate head or with distance-weighted binary loss, the result reduces to a cheaper recipe and we will report it as such. A secondary check sweeps $\tau$ and $\alpha$ to confirm the method is not fragile to these settings. We also train with targets computed from slightly eroded and dilated masks, to measure sensitivity to the annotation noise common in COCO polygons. Finally, we inspect the distractor subset. If performance there drops, the falling inside target is weakening the separation between the referent and look-alike objects. That would motivate keeping a contrastive term between objects alongside the regression term.