Rank-Limited Bilinear Fusion at Ambiguous Organ Boundaries

Medical segmentation networks fail most often at the edges of organs and lesions, not in their interiors. In a pancreas CT or a left-atrium MRI, the tissue on both sides of the true border often has nearly the same brightness. Whether a voxel is "pancreas" there depends on a conjunction: a faint local edge and surrounding anatomy indicating which side is which. Standard U-Nets join their high-resolution shallow features and their blurry but semantically rich deep features by concatenation or addition. Those operations can only add evidence. They cannot express "this edge matters only if the context says so." Factorized bilinear pooling is a way to multiply two feature vectors so that every pair of their entries can interact, while keeping the parameter count small through a low-rank factorization. It was built for exactly this kind of conjunction between two information sources, but only for one global vector per image in visual question answering.

We propose to move factorized bilinear fusion into the U-Net skip connection, compute it per voxel, and switch it on only inside the uncertain band around the predicted boundary. We also treat its rank as a capacity knob to be measured against training-set size. The claim is falsifiable. The gain from multiplicative skip fusion should sit almost entirely near low-contrast boundaries, should grow as the training set shrinks from about 80 scans toward 10, and should require rank greater than one. If rank one is as good as higher ranks, then an ordinary attention gate would do the same job and the bilinear machinery is not worth its cost. The experiments are designed to tell these outcomes apart.

Core Claim

Low-rank bilinear pooling has two main formulations. The low-rank bilinear model in Hadamard Product for Low-rank Bilinear Pooling projects both inputs and multiplies them element by element. The multimodal factorized version in Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering adds an expansion factor $k$ and a sum-pooling squeeze, so each output channel becomes a rank-$k$ bilinear form. Both supply our method, but both operate on a single pooled image vector and a question vector.

Our fusion is applied at each voxel $p$. Let $s(p) \in \mathbb{R}^{C_s}$ be the shallow skip feature and $g(p) \in \mathbb{R}^{C_g}$ the upsampled deep feature. Let $U \in \mathbb{R}^{C_s \times kO}$ and $V \in \mathbb{R}^{C_g \times kO}$ be learned projections, implemented as $1\times1$ convolutions. The interaction is

$$z(p) = \mathrm{SumPool}_k\big((U^\top s(p)) \odot (V^\top g(p))\big) \in \mathbb{R}^{O},$$

where $\odot$ is the element-wise product and $\mathrm{SumPool}_k$ sums non-overlapping groups of $k$ entries. Each output channel $o$ therefore equals $s(p)^\top W_o\, g(p)$ for a matrix $W_o$ of rank at most $k$.

We do not use this everywhere. Let $S$ be the coarse logit map from the previous decoder stage. A boundary-band weight $m(p) = 1 - |2\sigma(S(p)) - 1|$ is near one where that prediction is uncertain and near zero in confident interiors and background. The fused skip feature is

$$f(p) = \mathrm{Conv}_{1\times1}\big([\,s(p), g(p)\,]\big) + m(p)\, \phi\big(z(p)\big),$$

where $\phi$ is a normalization that stabilizes the product, and the first term is the ordinary concatenation path.

This design responds to three findings in the file:

  • Reverse attention. MISNet uses $1-\sigma(S)$ to send decoder capacity toward the uncertain zone. We use a symmetric uncertainty band instead, because the ambiguous voxels lie on both sides of the border.
  • Limits of additive feedback. In Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation, the recursive refinement feeds probabilities back by summation and is capped at two iterations. That paper does not test multiplicative modulation by the uncertainty map.
  • No gain from residual shortcuts. The low-rank bilinear paper reports that ResNet-style shortcuts around bilinear blocks did not help. We therefore keep the additive path as the main route and treat the bilinear term as a gated correction, rather than stacking bilinear blocks residually.

Research Program

The module. We build a single drop-in skip-fusion module for a 3D U-Net (and a 2D Res2Net variant for polyps). The backbone, losses, and augmentation are held fixed, so the only thing that changes is how the skip connection fuses its two inputs.

Normalization. The visual question answering recipe applies signed square root and then $\ell_2$ normalization to the whole fused vector, which MFB with co-attention found necessary to avoid exploding gradients. Applied per voxel, $\ell_2$ normalization would erase magnitude, and magnitude may carry edge strength. We therefore compare three choices of $\phi$: the original signed-square-root-plus-$\ell_2$ recipe, signed square root followed by group normalization, and group normalization alone. Whichever choice is stable and best is used for all later experiments.

Rank and regularization. We sweep the expansion factor $k \in \{1, 2, 5, 10\}$ at a fixed parameter budget. Towards Good Practices for Multi-modal Fusion in Large-Scale Video Classification reported that factorized bilinear fusion behaved as a regularizer, whereas parameter-matched linear fusion overfit. That evidence comes from 6.1 million videos, so it motivates the small-sample hypothesis but says nothing direct about 20 CT scans. The learning-curve experiments below test it in that regime.

Analysis by location. Every metric is reported separately for three regions: a thin band around the ground-truth surface, the interior, and the rest of the volume. Within the band, voxels are further split by local contrast, measured as the intensity difference across the true border. The prediction is specific: gains should appear in the low-contrast band and not in the interior.

Baselines

Naive reference. A plain 3D U-Net with concatenation skips, trained with Dice plus cross-entropy and the same augmentation. For context only, GGPFN reports a 3D U-Net at 81.13% mean Dice on MSD Pancreas, and the left-atrium paper reports 82.01% Dice for its cross-entropy baseline. Our own numbers on our splits are what count.

Parameter-matched concatenation. The same U-Net, with the skip $1\times1$ convolution widened, or deepened to two layers, until its parameters and floating-point operations match our module. This isolates multiplication from added capacity.

Attention gate, our strongest mechanistic competitor. A gate computed from both the shallow and deep features rescales the skip feature through a sigmoid. It is multiplicative but rank-limited, and it is width-matched to our module. MISNet's channel-attention fusion module is a published instance of this family.

Strongest published systems. - NIH Pancreas: DB-MSMUNet reports 89.47% Dice, and GGPFN reports 85.46%. - Left atrium 2018 (80 training, 20 test volumes): the full framework of Combating Uncertainty with Novel Losses reports 92.24% Dice. - ETIS polyps: MISNet reports only a 16.8% relative drop. No reported absolute result was found on Lacuna for ETIS mDice.

Published numbers come from different splits and pipelines, so we will not claim to beat them by citation. Instead we add our module to a reimplementation of the strongest available pipeline in each setting, on identical folds. On the left atrium, that means our module plus Overlap Loss and Focal Positive Loss, compared against the same pipeline without it.

Ablations. Two ablations isolate the key ingredients:

  • Band ablation: bilinear fusion applied everywhere ($m \equiv 1$) versus only in the band.
  • Rank ablation: $k = 1$ versus larger $k$.

Fairness protocol. Every method gets the same number of hyperparameter trials, about twenty, covering learning rate, weight decay, and width (plus $k$ for ours). Each runs under the same GPU-hour cap, with five seeds and three random draws of each training subset.

What counts as beating the baselines. A win is a statistically significant improvement in 95th-percentile Hausdorff distance and in boundary-band Dice over both the parameter-matched concatenation and the attention gate at training sizes of 40 scans or fewer, with no loss of interior Dice. Matching the published systems is a secondary target.

Why not just…?

Why not just widen the concatenation convolution? The video-classification study found that parameter-matched linear fusion overfit where factorized bilinear fusion did not. That was a large-data classification result, however, so it is suggestive rather than decisive here. The parameter-matched concatenation baseline, run along the learning curve, answers the question directly. If it closes the gap at small training sizes, the multiplicative claim fails.

Why not just use an attention gate? This is the real threat. A sigmoid gate formed from both features is already multiplicative, and at $k=1$ our interaction is close to a product of two scalar projections per output channel. The file's evidence that bilinear interaction beats attention comes from question answering, not segmentation. We therefore treat this as open. The rank sweep settles it: if $k=1$ matches $k=5$ and the gate matches both, we will report that multiplicative gating is sufficient and that the factorization adds nothing, which is itself a useful result for the field.

Why not just use boundary losses? On the left atrium, Overlap Loss and Focal Positive Loss raised Dice from 82.01% to 91.54% with no architectural change. Losses are therefore a strong, cheap alternative. We do not compete with them. We stack our module on top of them and ask whether the remaining boundary error shrinks. If the module helps only without the losses, it is redundant.

Why not just rely on pretraining and augmentation? Augmentation is held fixed and identical across all arms. The left-atrium pipeline's C3D initialization is reused where we reimplement it. Any gain therefore comes on top of these measures rather than in place of them.

Proposed Experiments

Learning curves. On NIH Pancreas (82 CT volumes) and the left-atrium challenge (80 training / 20 test MR volumes), we train with 10, 20, 40, and all available scans. Each subset size uses three random draws and five seeds. We report Dice, 95th-percentile Hausdorff distance, and average surface distance, each broken down by region and by contrast stratum.

First decisive check. The gap between our module and the parameter-matched concatenation in low-contrast boundary-band Dice must widen as the training set shrinks, while interior Dice stays flat. If gains are uniform across regions, the boundary-specific story is wrong, even if overall Dice improves.

Second decisive check. The band-restricted module must beat the version applied everywhere at small training sizes. This is the evidence that confining the interaction is what controls overfitting.

Third decisive check. Rank greater than one must beat both $k=1$ and the attention gate. Otherwise the bilinear framing should be dropped.

Transfer to unseen data. We train polyp models on Kvasir-SEG and CVC-ClinicDB and test them unchanged on ETIS, whose flat, concealed polyps have indistinct margins. We measure each method's relative drop against MISNet's reported 16.8%. Finally, a cross-site check trains on NIH Pancreas and tests on MSD Pancreas, asking whether boundary gains survive a change of scanner rather than reflecting site-specific texture.