Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation

Hongbo Zheng, Afshin Bozorgpour, Dorit Merhof, Tianyin Xu

Abstract

Medical image segmentation requires models that preserve fine anatomical boundaries while remaining efficient for clinical deployment. While transformers capture long-range dependencies, they suffer from quadratic attention cost and large data requirements, whereas CNNs are compute-friendly yet struggle with global reasoning. Linear attention offers $\mathcal{O}(N)$ scaling, but often exhibits training instability and attention dilution, yielding diffuse maps. We introduce PVT-GDLA, a decoder-centric Transformer that restores sharp, long-range dependencies at linear time. Its core, Gated Differential Linear Attention (GDLA), computes two kernelized attention paths on complementary query/key subspaces and subtracts them with a learnable, channel-wise scale to cancel common-mode noise and amplify relevant context. A lightweight, head-specific gate injects nonlinearity and input-adaptive sparsity, mitigating attention sink, and a parallel local token-mixing branch with depthwise convolution strengthens neighboring-token interactions, improving boundary fidelity, all while retaining $\mathcal{O}(N)$ complexity and low parameter overhead. Coupled with a pretrained Pyramid Vision Transformer (PVT) encoder, PVT-GDLA achieves state-of-the-art accuracy across CT, MRI, ultrasound, and dermoscopy benchmarks under equal training budgets, with comparable parameters but lower FLOPs than CNN-, Transformer-, hybrid-, and linear-attention baselines. PVT-GDLA provides a practical path to fast, scalable, high-fidelity medical segmentation in clinical environments and other resource-constrained settings.

Introduction

Medical image segmentation is a high-stakes balancing act requiring both global context—to understand organ relationships—and fine-grained local detail to trace precise boundaries of tumors or vessels. While Vision Transformers (ViTs) have revolutionized the field by capturing long-range dependencies, they impose a "quadratic tax": computational costs explode as image resolution increases ($O(N^2)$).

Linear attention offers a faster $O(N)$ alternative by swapping Softmax for kernel feature maps. However, it frequently suffers from "attention dilution," where non-negative kernels produce blurry, low-contrast maps that fail at anatomical boundaries. Additionally, these models often fall victim to "attention sinks," where the model incorrectly fixates on a few irrelevant tokens. PVT-GDLA is designed to bridge this gap, providing the speed of linear scaling with the precision of traditional Transformers.

Proposed Method: PVT-GDLA

The PVT-GDLA architecture pairs a pretrained Pyramid Vision Transformer (PVT) encoder with a specialized Gated Differential Linear Attention (GDLA) mixer in the decoder. The GDLA mixer addresses the limitations of linear attention through three primary innovations:

  1. Differential Subtraction: Inspired by differential signaling in electronics, the model splits queries and keys into two subspaces, calculates two separate attention paths, and subtracts one from the other. This cancels "common-mode noise" and amplifies the relevant signal, effectively sharpening the model's focus.
  2. Head-Specific Gating: A lightweight Sigmoid gate is added to each attention head. This injects nonlinearity and allows the model to ignore irrelevant areas through input-adaptive sparsity, directly mitigating the attention sink problem.
  3. Local Token Mixing: To maintain spatial awareness, a parallel branch uses depthwise convolutions (DWC). This reinforces spatial locality, which is crucial for high-fidelity boundary delineation.
Architecture of PVT-GDLA and the Gated Differential Linear Attention Mixer.

The model also utilizes a Mix-FFN (Feed-Forward Network) incorporating depthwise convolutions and SiLU nonlinearities to further enhance local feature extraction within the decoder blocks.

Experimental Evaluation

The authors evaluated PVT-GDLA across diverse medical imaging modalities, including the Synapse Multi-Organ CT (abdominal organs), ACDC (cardiac MRI), BUSI (breast ultrasound), and HAM10000/PH2 (dermoscopy).

Key Results

PVT-GDLA consistently outperformed both CNN-based and Transformer-based baselines:

  • Synapse CT: Achieved a 85.32 Dice score, surpassing the previous state-of-the-art (CENet at 85.04) while maintaining higher computational efficiency.
  • Computational Efficiency: The model operates at 6.85G FLOPs, significantly lower than many hybrid models like Swin-UNet or HiFormer, despite having comparable parameter counts (~32M).
  • Visual Fidelity: Qualitative results demonstrate that while standard linear attention produces diffuse activation "blobs," GDLA generates sharp, anatomically coherent masks that align precisely with organ boundaries.
Performance comparison: Dice score vs. parameters and FLOPs.

Discussion and Insights

The most significant insight is that subtraction acts as a sharpening tool. By subtracting a complementary subspace, the model removes the "background haze" inherent in linear attention, leaving only high-contrast features. Furthermore, the gating mechanism proves vital for stability; it prevents the model from collapsing onto a few tokens by forcing a more effective distribution of attention across the image.

PVT-GDLA demonstrates that the bottleneck in segmentation isn't necessarily parameter count, but the quality of attention focus. It achieves superior results to "heavy" models like TransUNet while using significantly fewer FLOPs, suggesting a more efficient path for clinical AI deployment.

Conclusion

PVT-GDLA offers a scalable solution for high-fidelity medical segmentation. While currently optimized for 2D benchmarks, future work will involve extending the GDLA mechanism to 3D kernels for native volumetric processing. The model's linear-time complexity and low FLOP count make it a strong candidate for deployment on edge-computing hardware in clinical settings.

Summary updated

Abstract

Medical image segmentation requires models that preserve fine anatomical boundaries while remaining efficient for clinical deployment. While transformers capture long-range dependencies, they suffer from quadratic attention cost and large data requirements, whereas CNNs are compute-friendly yet struggle with global reasoning. Linear attention offers $\mathcal{O}(N)$ scaling, but often exhibits training instability and attention dilution, yielding diffuse maps. We introduce PVT-GDLA, a decoder-centric Transformer that restores sharp, long-range dependencies at linear time. Its core, Gated Differential Linear Attention (GDLA), computes two kernelized attention paths on complementary query/key subspaces and subtracts them with a learnable, channel-wise scale to cancel common-mode noise and amplify relevant context. A lightweight, head-specific gate injects nonlinearity and input-adaptive sparsity, mitigating attention sink, and a parallel local token-mixing branch with depthwise convolution strengthens neighboring-token interactions, improving boundary fidelity, all while retaining $\mathcal{O}(N)$ complexity and low parameter overhead. Coupled with a pretrained Pyramid Vision Transformer (PVT) encoder, PVT-GDLA achieves state-of-the-art accuracy across CT, MRI, ultrasound, and dermoscopy benchmarks under equal training budgets, with comparable parameters but lower FLOPs than CNN-, Transformer-, hybrid-, and linear-attention baselines. PVT-GDLA provides a practical path to fast, scalable, high-fidelity medical segmentation in clinical environments and other resource-constrained settings.
BibTeX
@misc{zheng2026gated,
  author = {Hongbo Zheng and Afshin Bozorgpour and Dorit Merhof and Tianyin Xu},
  title = {{Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation}},
  year = {2026},
  eprint = {2603.02727},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02727},
}
Figure