Segmentation networks for medical images usually get lesion centers right and edges wrong. At a low-contrast margin, such as a faint ground-glass opacity fading into healthy lung, the pixels on either side look almost the same. A standard loss rewards the easy interior as much as the hard perimeter, so the model blurs the edge. One long-standing fix is reverse attention: take the model's confident first-pass mask, erase it from the feature maps, and force later layers to look only at what is left. This is purely visual. It cannot tell a faint true lesion margin from a benign fold of tissue that looks equally faint. Clinical text, such as the short radiology notes paired with chest X-rays and CT slices, often says exactly what the image cannot: "bilateral infection, lower zone of the left lung."
We propose to apply the erasure idea to both modalities at once. At each refinement stage, the model erases what its current mask already explains in the image, as usual. It also down-weights the words of the report that the current mask already accounts for. The remaining, "unexplained" words then direct attention onto the remaining uncertain pixels, and the evidence in those pixels in turn updates the words. In two sentences: the report tells the model what it has not yet found, and the ambiguous boundary pixels tell the report which of its claims are still unresolved. Both sides shrink together until the mask accounts for the text.
Core Claim
In low-contrast, text-paired segmentation, the text helps most where the visual evidence is weakest: at the ambiguous edge and in small missed regions. It helps there only if it is used in two ways. It must be re-grounded in the leftover uncertain region rather than in the whole image. It must also be filtered to the words that the current prediction has not yet satisfied. We expect a model that performs this two-sided erasure to beat three alternatives on boundary distance and small-lesion Dice. The first is text-guided models without any reverse attention. The second is reverse-attention models without text. The third, and most important, is a model that simply stacks the two, where text conditions the network but the reverse-attention stage never sees it. The same design also makes high-resolution cross-attention affordable, because attention is computed only over the thin band of uncertain pixels instead of the full grid.
Where This Sits in Prior Work
Four lines of prior work motivate the proposal, supply its methods, or set the bar it must clear.
Two papers motivate it. Reverse attention was introduced in PraNet, which erases the predicted foreground as "one minus the mask" so that residual layers focus on transition zones. PraNet-V2 showed that this rule-based inversion is fragile. It replaced the inversion with separate supervised foreground and background heads and a learnable gain, and its authors name text-guided prompting as future work. We take up that suggestion, but our design differs in one respect: the text is not a static prompt. It is itself subject to erasure.
A second group, language-guided medical segmentation, supplies the bidirectional machinery and exposes the gap. LViT injects report embeddings into visual features, mainly in one direction, text to image. RecLMIS makes the exchange two-way through masked reciprocal reconstruction of image patches and words. Its authors report that it fails on small targets in MosMedData+, which is consistent with cross-attention at coarse patch resolution averaging a tiny lesion with its surroundings. FMISeg runs two-way attention between report features and wavelet high- and low-frequency features, and adds a filter that suppresses lesion-like regions not mentioned in the note.
The closest existing work is FMISeg combined with PraNet-V2. FMISeg already uses text to veto visually plausible false regions, which is one half of our argument. It does this once, over the whole feature map, and it never asks which words remain unexplained by the current mask. Neither FMISeg nor RecLMIS iteratively erases confident regions. PraNet-V2 erases, but it has no text.
In natural images, CEFNet pairs encoder co-attention with a boundary module that subtracts high-level features to expose edges. Its boundary module is visual only and never feeds back into the language features, and it was not tested on low-contrast data. Its ablation, a roughly 3% IoU gain from fusing text early rather than only in the decoder, is evidence that where text enters the network matters. We reuse that lesson by letting text enter at every refinement stage.
Finally, two constraints shape the design. Boundary-specific regularizers such as the key-point and shape evaluator of Structure Boundary Preserving Segmentation and the contour loss of Multi-scale Features and Contour Loss Constrain give consistent but small gains; for example, the former improves Dice on the ultrasound dataset from 82.40% to 83.52%. This warns that boundary machinery alone moves region metrics little. We will therefore measure boundary-specific metrics, not only Dice. The synonym sensitivity reported by RecLMIS also constrains us: a mechanism that keys on individual words must be tested under paraphrase.
Research Program
The base network is a hybrid encoder and decoder of the LViT/FMISeg family with a frozen clinical text encoder. On top of it, we build $S$ refinement stages running from coarse to fine resolution.
At stage $s$, the previous mask probability $p_i$ at pixel $i$ defines an uncertainty band weight:
$$b_i = 1 - |2p_i - 1|.$$
This weight is 1 where the model is undecided and 0 where it is confident. Following PraNet-V2, we use its supervised foreground and background heads and widen the band wherever the two heads disagree. Only the top-$K$ band pixels at the current resolution become attention tokens. This caps cost at $K$ times the number of words rather than the full grid, which is what makes attention at small-lesion resolution affordable.
Each word is also re-weighted by how much of its grounding the current mask already covers. We call the result the word residual weight:
$$r_j = 1 - \frac{\sum_i p_i\, a_{ij}}{\sum_i a_{ij}}.$$
Here $a_{ij}$ is the attention weight between pixel $i$ and word $j$ from the previous stage. A word such as "left" whose attention already falls inside the predicted mask gets $r_j$ near 0. A word whose attention lies mostly outside the mask stays active.
The stage then runs two cross-attention passes. In the first, residual-weighted words attend to the band tokens, which updates the words with evidence from the uncertain region. In the second, the band tokens attend back to the updated words and predict a foreground or background correction for those pixels, which is added to the mask.
A light auxiliary loss enforces the intended behaviour. At the final stage, the residual weights of words that correspond to annotated lesion regions should be driven toward zero. We train this against pseudo-groundings computed from ground-truth masks and the attention maps, so no extra labels are needed.
The analysis has three parts. The first is attention and residual-weight visualizations on cases where the first-pass mask misses a lesion mentioned in the report. The second is stratified evaluation by lesion area and by local contrast at the boundary. The third is counterfactual text edits, described below, which test whether the model is using the words or merely benefiting from extra capacity.
Baselines
The naive reference points are a vision-only U-Net and nnUNet trained on the same splits. For nnUNet, LViT reports 80.42% Dice on QaTa-COV19. These show how much the text helps at all.
The strongest existing method is FMISeg. It reports 91.21% Dice on QaTa-COV19 and 79.30% Dice on MosMedData+ (FMISeg). It is the right main competitor because it is two-way, text-guided, and explicitly aimed at blurry boundaries. We will retrain it from public code on identical splits, backbone, input resolution, and epoch and hyperparameter search budget, and report both our reproduction and the published figure. For boundary behaviour we also include RecLMIS, for which no absolute Dice or mIoU is reported on Lacuna; only margins over LViT of 1.89% mIoU on QaTa-COV19 and 3.74% mIoU on MosMedData+ are available. We also include LViT itself, which reports 83.66% Dice on QaTa-COV19. The vision-only reverse-attention reference is PraNet-V2's supervised foreground-background reverse attention on the same backbone, with no text.
The decisive baseline is a stacked model: FMISeg with PraNet-V2 reverse-attention stages appended, where text conditions the encoder and decoder but not the refinement stages. It has matched parameters and matched band resolution. Three ablations then isolate our key ingredients. The first is one-way text-conditioned refinement, in which words attend to nothing and simply condition the band. The second is two-way refinement without word residual weighting. The third is full-grid attention in place of band-restricted attention at equal compute.
Beating FMISeg on overall Dice alone would not be enough, since its QaTa number leaves little headroom. The proposal succeeds only if the full model beats both FMISeg and the stacked model on HD95 and on Dice for the smallest lesion-area tercile of MosMedData+, by more than the seed-to-seed standard deviation across five seeds, while matching them on overall Dice.
Why not just…?
Why not just bolt PraNet-V2's reverse attention onto FMISeg? This is the real threat, and the stacked baseline exists to settle it. Our hypothesis is that erasure without text re-grounding still cannot separate a faint lesion margin from faint benign structure, because text enters the network before the ambiguous band is defined. If the stacked model matches ours on small-lesion Dice and HD95, the two-way and residual-word parts add nothing. In that case we would report the simpler stack as the result.
Why not just let the text condition the refinement one way, without updating the words? One-way conditioning feeds the whole report to every stage, including words such as "right lung" that are already satisfied. That dilutes the signal pointing to missed regions. CEFNet's gain from earlier, deeper fusion and RecLMIS's gain over the more one-directional LViT both suggest that the direction and placement of the exchange matter. Neither tested it inside a refinement loop, however, so the one-way ablation is the direct test.
Why not just use a vision-only boundary loss or a stronger backbone? Boundary losses give small region-metric gains (Structure Boundary Preserving Segmentation). Vision-only models also collapse on hard, subtle cases: PraNet reaches only 0.628 mDice on ETIS, and competitors drop by more than 38% under domain shift (MISNet). The gaps between text-guided models and nnUNet reported by LViT and FMISeg also argue against this route. Still, we include a vision-only model with a contour loss at matched capacity, because part of any text-model gain may come from capacity rather than from the words.
Why not just prompt-tune a foundation segmenter? CaPro reports that generic foundation-model features fail on thin, low-contrast structures without specialized prompting. Progressive Multi-modal Conditional Prompt Tuning argues that single-pass prompting misses iterative mutual adjustment. We treat a SAM-style model with report-derived prompts as a secondary reference rather than a main competitor.
Proposed Experiments
The primary benchmarks are QaTa-COV19 and MosMedData+, the text-paired datasets in the file. MosMedData+ is the decisive regime because of its small, low-contrast infections. ESO-CT, used by LViT, is a secondary test on narrow tubular targets; no reported number was found on Lacuna for it. The polyp benchmarks (Kvasir-SEG, CVC-ClinicDB, ETIS) have no reports. We will not fabricate text for them, so claims stay limited to text-paired data. Metrics are Dice, mIoU, HD95, boundary F-score within a fixed pixel tolerance, and Dice stratified by lesion area and by boundary contrast.
Three checks decide the question. The first is the small-lesion test: the full model must beat both FMISeg and the stacked baseline on the smallest-area tercile of MosMedData+ and on HD95.
The second is a set of text interventions. We shuffle reports across patients, which should erase the gain. We delete a location word such as "left," which should remove predictions in the corresponding region. We substitute clinical synonyms, which should leave the mask nearly unchanged. If the gain survives shuffling, the model is not reading the text. If it collapses under synonyms, it has memorized tokens, which is the weakness RecLMIS reported.
The third is a residual-word diagnostic. On cases where the first stage misses a reported lesion, the residual weight of the corresponding word should fall only after the mask covers that region. We will measure the correlation between a word's residual-weight drop and the recovered lesion area. A null correlation would mean the mechanism works for reasons other than the one claimed, and we would say so.
References
- PraNet: Parallel Reverse Attention Network for Polyp Segmentation
- PraNet-V2: Dual-Supervised Reverse Attention for Medical Image Segmentation
- LViT: Language meets Vision Transformer in Medical Image Segmentation
- Cross-Modal Conditioned Reconstruction for Language-Guided Medical Image Segmentation
- Frequency-Domain Multi-modal Fusion for Language-Guided Medical Image Segmentation
- Encoder Fusion Network With Co-Attention Embedding for Referring Image Segmentation
- Structure Boundary Preserving Segmentation for Medical Image With Ambiguous Boundary
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- CaPro: Curvilinear-aware Prompt Learning with Single Unlabeled Image for Cost-effective Curvilinear Structure Segmentation
- Progressive Multi-modal Conditional Prompt Tuning