Letting Each MRI Sequence Vote on the Boundary It Sees Best

Brain tumor MRI arrives as several co-registered scans called sequences (T1, contrast-enhanced T1c, T2, FLAIR), and a tumor margin that is invisible in one sequence is often sharp in another. Current segmentation models waste this. Networks that care about boundaries add an edge-detection branch but read only one image. Networks that fuse sequences use cross-modal attention, a mechanism where features from one input decide which parts of another input to read. They apply it uniformly over the whole image and never learn edges explicitly. We propose a module that does two things. It spends cross-modal attention only on the thin band of voxels where the model is still unsure about the boundary. Within that band, a learned per-location gate decides which sequence to trust for the edge at that spot.

The idea in two sentences: first make a coarse fused segmentation. Then, only at its uncertain rim, let each voxel query every sequence's edge features and weight them by how much that sequence can be trusted there. We expect the payoff to concentrate on cases where sequences disagree about where the border is. The central experiment tests exactly that.

Core Claim

Our claim is narrower than "boundary branches plus cross-attention help." We claim that boundary errors in multi-sequence segmentation are driven by location-dependent asymmetry in which sequence shows the edge. We further claim that a mechanism resolving this asymmetry locally, restricted to the uncertain rim, beats three weaker alternatives:

  • early concatenation into a boundary-aware network;
  • dense cross-modal attention with an added boundary loss;
  • a global, per-sequence weighting that ignores location.

If concatenation with a boundary loss matches us, the claim is false. We have designed the experiments so that outcome is visible.

Relation to Prior Work

Two bodies of work each supply half the ingredients and motivate the gap.

Boundary-aware medical segmentation adds explicit edge supervision to single images:

All three read one modality, so they cannot choose a better view of an edge that is faint in the one they have.

On the fusion side, Multi-Modal Medical Image Segmentation using Vision Transformers (ViTs) (ViT-Med) gives each sequence its own encoder and fuses them with dense cross-modal attention. It reports Dice 0.91 and HD95 3.1 mm on BraTS whole tumor, against 0.85 and 5.1 mm for a U-Net. It has no boundary branch or edge loss, and its dense attention cost is what keeps it on 2D or pseudo-3D patches.

We borrow mechanisms from three further papers:

The closest existing system is ESEG. It fuses sparse event-camera edges with dense RGB semantics through density-aware dynamic-window cross-attention and gains 1.7% mIoU over SegFormer on DSEC-Semantic. It differs from our proposal in three ways:

  • Its attention windows are set by event density, not by the model's own boundary uncertainty.
  • It fuses two streams rather than choosing among several.
  • Its edges come from SAM pseudo-labels.

Our boundary targets come from the ground-truth masks, and the question we test, which sequence shows this edge, has no analogue in its setting.

One constraint on the design comes from the Sobel-based work. Fixed gradient operators amplify speckle and clutter when edges are not sharp steps, as the E2Net and Structure Boundary Preserving Segmentation results suggest. We therefore use learned per-sequence edge features rather than fixed operators, and keep a Sobel variant only as an ablation.

Research Program

Encoders and coarse prediction. Each sequence $m$ passes through its own encoder, as in ViT-Med. Each encoder has a small auxiliary edge head supervised by the boundary of the ground-truth mask, using the distance-weighted loss style of E2Net to counter the rarity of edge voxels. A coarse fused decoder produces a foreground probability $p$.

The rim. The uncertain band is the set of voxels $B = \{x : |p(x) - 0.5| < \tau\}$, dilated by a few voxels, where $\tau$ is a threshold. This set plays the role that erased foreground plays in reverse attention.

Rim refinement. For each rim voxel $x$, with fused query feature $q_x$, we attend within a local window $W(x)$ into each sequence's edge-aware features $k^m, v^m$. We then combine the sequences with a softmax gate $g^m(x)$ predicted from $q_x$:

$$ h_x = q_x + \sum_{m} g^m(x)\,\sum_{y \in W(x)} \operatorname{softmax}_y\!\left(\frac{q_x^\top k^m_y}{\sqrt{d}}\right) v^m_y,\qquad \sum_m g^m(x)=1 . $$

Here $d$ is the feature dimension. Because attention runs only over $|B|$ rim voxels and windows of fixed size $|W|$, cost scales as $|B|\cdot|W|$ times the number of sequences rather than quadratically in volume size. This is what we hope makes native 3D feasible. The refined rim features replace the coarse ones, and a final head predicts the mask with Dice plus boundary loss.

Measuring asymmetry. We build an analysis tool that is also the basis of the decisive test. For every ground-truth boundary voxel, we compute each sequence's local contrast across the boundary normal. From these we derive two quantities:

  • a per-voxel visibility profile across sequences;
  • a per-case asymmetry index: the average spread between the most- and least-visible sequence.

This lets us ask whether the learned gate tracks visibility, and whether our gains concentrate in high-asymmetry cases. We keep an optional weak auxiliary loss that nudges the gate toward the visibility profile, reported as a separate variant.

Baselines

All baselines are reimplemented in one codebase. They share:

  • the same BraTS training and test split;
  • the same preprocessing and augmentation;
  • an equal hyperparameter-search budget (same number of trials per method);
  • matched parameter counts within roughly 10%;
  • the same number of training epochs.

Published numbers are quoted only as sanity checks, because the splits and settings differ. ViT-Med reports whole-tumor Dice 0.91 and HD95 3.1 mm, and Boundary Aware Networks reports tumor-core Dice 0.822 and Hausdorff 3.406 on 2D T1c.

Naive reference. A U-Net on concatenated sequences with Dice loss only. ViT-Med reports this family at Dice 0.85 and HD95 5.1 mm on BraTS.

Concatenation plus boundary supervision. The Boundary Aware Networks shape stream and edge loss, fed all four sequences as input channels. This is the strongest simple competitor, because it tests whether explicit edges plus access to all sequences already suffice without any learned choice among them.

Strongest existing fusion method. ViT-Med-style dense cross-modal attention. We run it twice: as published, and with our edge heads and boundary loss added. The second run is the obvious combination of two existing ideas and is the bar we must clear.

Ablations isolating our ingredients. Each changes one piece of our model:

  • Global gate: the gate is a per-sequence scalar for the whole image, not per location.
  • Everywhere attention: the rim restriction is removed and windowed attention runs over all voxels.
  • Sobel edges: the learned edge features are swapped for Sobel features, as in CTO.
  • No gate: sequences are averaged uniformly.

What counts as winning. We beat the field if we improve boundary metrics (HD95 and boundary F-score at a 1–2 voxel tolerance) over the boundary-augmented ViT-Med and the concatenation baseline. The improvement must be consistent across five seeds, and region Dice must not fall.

Why not just…?

Why not just concatenate all sequences into an existing boundary-aware network? This is the real threat. ViT-Med's advantage over a concatenating U-Net (0.85 vs 0.91 Dice) confounds fusion style with the missing edge supervision, so it does not settle the question. Our concatenation-plus-boundary-loss baseline settles it directly. If that baseline matches us overall and also in the high-asymmetry tertile of cases, we drop the gating claim and report that explicit edges are what matter.

Why not just add a boundary loss to dense cross-modal attention? Dense attention can in principle learn to favor the right sequence per location. However, An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention argues that high-magnitude inputs can dominate ungated attention, and dense cost limits resolution. The boundary-augmented ViT-Med baseline tests this at matched compute. We also report the memory it needs at full 3D resolution. If it wins at equal memory, our rim restriction is only an efficiency trick, and we will say so.

Why not just learn a global weight per sequence, or clean up edges afterward? A global weight captures "FLAIR is generally more useful" but not "T1c shows this margin while FLAIR shows that one." The global-gate ablation measures exactly the value of location dependence. As for post-processing, polarization thresholds as in the Res2Net reverse-attention work, and largest-connected-component filtering as noted in Enhancing Cross-Modal Medical Image Segmentation Through Compositionality, remove spurious blobs but cannot move a misplaced border. We apply the same post-processing to every method, so it cannot explain differences.

Proposed Experiments and Decisive Checks

Main benchmark. BraTS, with its three regions (whole tumor, tumor core, enhancing tumor), evaluated by Dice, HD95 and boundary F-score.

The decisive test. We stratify test cases by the asymmetry index. The hypothesis predicts that our margin over the concatenation and boundary-augmented ViT-Med baselines grows with asymmetry and is near zero in the lowest tertile. A flat margin across tertiles would mean the gain comes from something other than resolving asymmetry.

Gate interpretability. We measure rank correlation between the learned gate and the per-voxel visibility profile, with and without the auxiliary visibility loss.

Missing-sequence robustness. At test time we drop one sequence by zeroing its gate and renormalizing. We compare how gracefully Dice and HD95 degrade against the baselines, which must drop the channel instead.

Second medical setting. CHAOS MRI (T1 in-phase and out-of-phase, T2) for liver. No reported result found on Lacuna for ViT-Med's CHAOS numbers, so there we compare only against our own reimplementations.

Transfer to event cameras. DSEC-Semantic with RGB and events as two sequences, reporting mIoU against SegFormer and ESEG. The reference point is ESEG's reported +1.7% mIoU over SegFormer. This checks whether rim-restricted gated attention helps outside medicine, where one stream is sparse.

Efficiency. We report memory and runtime at full 3D resolution for our model against dense attention. This shows whether the rim restriction enables native 3D, or whether the method must fall back to patches like ViT-Med.