TL;DR
The paper introduces Gated Differential Linear Attention (GDLA), an efficient $\mathcal{O}(N)$ decoder module designed to fix the "attention dilution" problem in linear attention mechanisms. By combining a dual-branch differential subtraction method with local token mixing and data-dependent gating, the proposed PVT-GDLA model achieves state-of-the-art medical image segmentation accuracy across multiple modalities while maintaining a highly favorable performance-to-compute ratio.
What Problem Does This Paper Address?
High-fidelity medical image segmentation demands a difficult architectural balancing act. Models must capture long-range global context (to understand macro-anatomical relationships) while preserving sharp, fine-grained local boundaries (for small or thin structures like blood vessels or specific organs).
Standard Vision Transformers (ViTs) excel at global reasoning but suffer from quadratic computational complexity $\mathcal{O}(N^2)$, making them too resource-intensive for high-resolution clinical deployment. Linear attention variants reduce this complexity to $\mathcal{O}(N)$ by swapping out the softmax operation for kernel feature maps. However, because these kernels are strictly non-negative, they tend to produce overly smooth, diffuse feature aggregations—a flaw known as attention dilution. This dilution blurs anatomical boundaries and degrades segmentation performance.
Methods
The authors propose the Gated Differential Linear Attention (GDLA) mixer, which they integrate into the decoding stages of an encoder-decoder network powered by a pretrained Pyramid Vision Transformer (PVT) encoder.
The GDLA block splits the feature processing into two parallel paths:
- Global Path (Gated Differential Linear Attention): To counter attention dilution while preserving $\mathcal{O}(N)$ scaling, the module projects the input into queries and keys, splitting them into two complementary channel subspaces. It computes two separate kernelized linear attention maps and combines them via learnable channel-wise subtraction ($\boldsymbol{A}_1 - \boldsymbol{\lambda} \odot \boldsymbol{A}_2$). This subtraction cancels out low-frequency noise and sharpens focus. Finally, a data-dependent SiLU gating mechanism modulates the output to adaptively refine tokens and suppress "attention sinks."
- Local Path (Local Token Mixing): Operating in parallel to the global path, this lightweight branch uses a $3\times3$ depthwise convolution (DWC) followed by a pointwise convolution to explicitly enforce neighborhood interactions and spatial continuity.
The outputs of both paths are concatenated and projected back to the model's dimensions. Additionally, the decoder block utilizes a customized Mix-FFN (Feed-Forward Network) that embeds depthwise convolutions directly within the multi-layer perceptron to further reinforce local boundary details.
Datasets
The architecture was rigorously evaluated across five public 2D medical image segmentation datasets spanning four distinct imaging modalities:
- Synapse Multi-Organ (Abdominal CT scans)
- Automated Cardiac Diagnosis Challenge (ACDC) (Cardiac MRI)
- Breast Ultrasound Images Dataset (BUSI) (Ultrasound)
- HAM10000 (Dermoscopic skin lesion images)
- PH$^2$ (Dermoscopic skin lesion images)
No new datasets were introduced in this study.
Key Insights and Ideas
- Preserving Associativity in Differential Attention: Standard differential attention subtracts attention matrices before multiplying by the value matrix ($\boldsymbol{V}$), a sequence that breaks the associative property required to achieve linear $\mathcal{O}(N)$ time. GDLA cleverly bypasses this by computing two independent, normalized linear attention outputs and subtracting them after the value aggregation, successfully bringing the contrast-enhancing benefits of differential attention to linear-time architectures.
- Mitigating Attention Sinks: Linear attention often suffers from an "attention sink" phenomenon where the model dumps disproportionate feature energy into the very first token, leading to a smeared, low-contrast feature map. As shown via channel saliency mapping, GDLA's gating and subtraction mechanisms redistribute feature energy more uniformly across relevant anatomical structures.
- Parallel Global-Local Synergy: Rather than processing global context and local details sequentially, executing them in parallel branches and fusing them at the block's end allows the model to retain sharp boundary definitions without sacrificing global coherence.
Key Results
PVT-GDLA consistently outperformed both traditional CNNs and heavy Transformer-based models across all benchmarks while demonstrating superior computational efficiency.
- Synapse (CT): PVT-GDLA achieved a state-of-the-art average Dice Similarity Coefficient (DSC) of 85.32% and slashed the 95th-percentile Hausdorff Distance (HD95) down to 12.41 mm—a massive improvement over baseline models like TransUNet (77.49% DSC / 31.69 mm HD95) and Swin-UNet (79.13% DSC / 21.55 mm HD95), despite using a fraction of the FLOPs.
- ACDC (MRI): The model achieved a top-tier average Dice score of 92.53%, excelling uniformly across right ventricle, myocardium, and left ventricle segmentations.
- BUSI (Ultrasound): It attained an average Dice score of 80.54%, outperforming its closest PVT-based baseline (PVT-EMCAD-B2) by +0.29% and standard linear attention by +1.40%.
- Dermoscopy: PVT-GDLA reached 95.59% Dice on PH$^2$ and 95.01% on HAM10000, proving its robust generalization capabilities on highly irregular boundaries.
Limitations and Future Work
Not discussed in the paper.
Summary updated