Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation

Hongbo Zheng, Afshin Bozorgpour, Dorit Merhof, Minjia Zhang

TL;DR

The paper introduces Gated Differential Linear Attention (GDLA), an efficient $\mathcal{O}(N)$ decoder module designed to fix the "attention dilution" problem in linear attention mechanisms. By combining a dual-branch differential subtraction method with local token mixing and data-dependent gating, the proposed PVT-GDLA model achieves state-of-the-art medical image segmentation accuracy across multiple modalities while maintaining a highly favorable performance-to-compute ratio.

What Problem Does This Paper Address?

High-fidelity medical image segmentation demands a difficult architectural balancing act. Models must capture long-range global context (to understand macro-anatomical relationships) while preserving sharp, fine-grained local boundaries (for small or thin structures like blood vessels or specific organs).

Standard Vision Transformers (ViTs) excel at global reasoning but suffer from quadratic computational complexity $\mathcal{O}(N^2)$, making them too resource-intensive for high-resolution clinical deployment. Linear attention variants reduce this complexity to $\mathcal{O}(N)$ by swapping out the softmax operation for kernel feature maps. However, because these kernels are strictly non-negative, they tend to produce overly smooth, diffuse feature aggregations—a flaw known as attention dilution. This dilution blurs anatomical boundaries and degrades segmentation performance.

Methods

The authors propose the Gated Differential Linear Attention (GDLA) mixer, which they integrate into the decoding stages of an encoder-decoder network powered by a pretrained Pyramid Vision Transformer (PVT) encoder.

Figure 2. PVT-GDLA overview. A pretrained Pyramid Vision Transformer (PVT) encoder extracts multi-scale features that are progres- sively decoded by GDLA blocks. Each GDLA block contains a GDLA mixer and a feed-forward network (FFN). Within the mixer, a gated differential linear-attention path captures globally selective context, while a parallel local token-mixing branch reinforces neighborhood interactions. The fused features are projected and upsampled with skip connections to recover spatial resolution.

The GDLA block splits the feature processing into two parallel paths:

  1. Global Path (Gated Differential Linear Attention): To counter attention dilution while preserving $\mathcal{O}(N)$ scaling, the module projects the input into queries and keys, splitting them into two complementary channel subspaces. It computes two separate kernelized linear attention maps and combines them via learnable channel-wise subtraction ($\boldsymbol{A}_1 - \boldsymbol{\lambda} \odot \boldsymbol{A}_2$). This subtraction cancels out low-frequency noise and sharpens focus. Finally, a data-dependent SiLU gating mechanism modulates the output to adaptively refine tokens and suppress "attention sinks."
  2. Local Path (Local Token Mixing): Operating in parallel to the global path, this lightweight branch uses a $3\times3$ depthwise convolution (DWC) followed by a pointwise convolution to explicitly enforce neighborhood interactions and spatial continuity.

The outputs of both paths are concatenated and projected back to the model's dimensions. Additionally, the decoder block utilizes a customized Mix-FFN (Feed-Forward Network) that embeds depthwise convolutions directly within the multi-layer perceptron to further reinforce local boundary details.

Datasets

The architecture was rigorously evaluated across five public 2D medical image segmentation datasets spanning four distinct imaging modalities:

  • Synapse Multi-Organ (Abdominal CT scans)
  • Automated Cardiac Diagnosis Challenge (ACDC) (Cardiac MRI)
  • Breast Ultrasound Images Dataset (BUSI) (Ultrasound)
  • HAM10000 (Dermoscopic skin lesion images)
  • PH$^2$ (Dermoscopic skin lesion images)

No new datasets were introduced in this study.

Key Insights and Ideas

  • Preserving Associativity in Differential Attention: Standard differential attention subtracts attention matrices before multiplying by the value matrix ($\boldsymbol{V}$), a sequence that breaks the associative property required to achieve linear $\mathcal{O}(N)$ time. GDLA cleverly bypasses this by computing two independent, normalized linear attention outputs and subtracting them after the value aggregation, successfully bringing the contrast-enhancing benefits of differential attention to linear-time architectures.
  • Mitigating Attention Sinks: Linear attention often suffers from an "attention sink" phenomenon where the model dumps disproportionate feature energy into the very first token, leading to a smeared, low-contrast feature map. As shown via channel saliency mapping, GDLA's gating and subtraction mechanisms redistribute feature energy more uniformly across relevant anatomical structures.
  • Parallel Global-Local Synergy: Rather than processing global context and local details sequentially, executing them in parallel branches and fusing them at the block's end allows the model to retain sharp boundary definitions without sacrificing global coherence.

Key Results

PVT-GDLA consistently outperformed both traditional CNNs and heavy Transformer-based models across all benchmarks while demonstrating superior computational efficiency.

Figure 1. Comparison of model performance on Synapse dataset and compute–accuracy trade-off. Our approach achieves the highest Dice score with lower # parameters and lower # FLOPs.
  • Synapse (CT): PVT-GDLA achieved a state-of-the-art average Dice Similarity Coefficient (DSC) of 85.32% and slashed the 95th-percentile Hausdorff Distance (HD95) down to 12.41 mm—a massive improvement over baseline models like TransUNet (77.49% DSC / 31.69 mm HD95) and Swin-UNet (79.13% DSC / 21.55 mm HD95), despite using a fraction of the FLOPs.
  • ACDC (MRI): The model achieved a top-tier average Dice score of 92.53%, excelling uniformly across right ventricle, myocardium, and left ventricle segmentations.
  • BUSI (Ultrasound): It attained an average Dice score of 80.54%, outperforming its closest PVT-based baseline (PVT-EMCAD-B2) by +0.29% and standard linear attention by +1.40%.
  • Dermoscopy: PVT-GDLA reached 95.59% Dice on PH$^2$ and 95.01% on HAM10000, proving its robust generalization capabilities on highly irregular boundaries.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Medical image segmentation requires models that preserve fine anatomical boundaries while remaining practical for clinical deployment. Transformers capture long-range dependencies but incur quadratic attention cost, whereas CNNs are efficient but less effective at global reasoning. Linear attention offers \(\mathcal{O}(N)\) scaling, but often produces diffuse feature aggregation that weakens boundary-sensitive prediction. We introduce a gated differential linear-attention mixer for medical image segmentation. Its global path, Gated Differential Linear Attention (GDLA), performs differential subtraction between two kernelized attention branches over complementary query/key subspaces to suppress redundant responses, and employs a data-dependent gate for token refinement. A parallel local token-mixing branch with depthwise convolution strengthens neighborhood interactions for better refinement, and the two branches are fused while preserving \(\mathcal{O}(N)\) complexity. When instantiated in a pretrained Pyramid Vision Transformer (PVT)-based encoder--decoder model, \name achieves state-of-the-art results on the evaluated 2D medical segmentation benchmarks spanning CT, MRI, ultrasound, and dermoscopy, with a favorable accuracy--efficiency trade-off over closely related baselines. The code is publicly available at \href{https://github.com/xmindflow/gdla}{https://github.com/xmindflow/gdla}.
BibTeX
@misc{zheng2026gated,
  author = {Hongbo Zheng and Afshin Bozorgpour and Dorit Merhof and Minjia Zhang},
  title = {{Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation}},
  year = {2026},
  eprint = {2603.02727},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02727},
}
Figure