L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation

Vandan Gorade, Sparsh Mittal, Neethi Dasu, Rekha Singhal, Kc Santosh, Debesh JhaarXiv 2025 · DOI · arXiv

Abstract

Medical image segmentation requires a precise balance between capturing sharp local boundaries and understanding global anatomical arrangements. While Transformers are popular for global context, they suffer from high computational complexity and sensitivity to redundant image regions. We present L2GNet, a novel architecture that replaces standard self-attention with an Optimal Transport-based mechanism called L2GMapper. By aligning discrete anatomical features with learnable reference codes using the Sinkhorn algorithm, L2GNet captures long-range dependencies with linear complexity. Experimental results on the Synapse and ACDC benchmarks demonstrate that L2GNet outperforms state-of-the-art Transformer-based models in both accuracy and boundary preservation.

Introduction

Accurate identification of organs in medical imaging is a "high-stakes balancing act." Models must discern fine-grained local details while maintaining a coherent global view of anatomy. Current approaches typically utilize either Continuous Latent Spaces (CLS), which often struggle with computational redundancy in Transformers, or Discrete Latent Spaces (DLS), which excel at local texture but fail to capture long-range relationships. Hybrid synergistic models attempt to bridge this gap but remain limited by the $O(n^2)$ complexity of self-attention. L2GNet addresses these challenges by rethinking global context through the lens of Optimal Transport (OT).

Methods: The L2GMapper

The core innovation of L2GNet is the L2GMapper bottleneck. Unlike standard self-attention that performs exhaustive pixel-to-pixel comparisons, L2GMapper utilizes Optimal Transport to align local features with a global anatomical template.

Architecture of L2GNet featuring the L2GMapper bottleneck.

The process involves four key steps:

  1. Discrete Coding: Encoder features are quantized into discrete codes, providing a structured, denoised representation of local anatomy.
  2. Sinkhorn Alignment: These codes are mapped to a Reproducing Kernel Hilbert Space (RKHS). The Sinkhorn algorithm then finds the optimal alignment between input codes and a set of trainable reference codes.
  3. Weighted Pooling: Features are pooled based on the transport plan, effectively grouping similar anatomical structures that match the learned references.
  4. Linear Efficiency: By using a fixed number of reference codes, the mechanism achieves linear complexity relative to input size, offering significant memory advantages over Transformers.

Results and Discussion

L2GNet was evaluated on the Synapse Multi-organ Segmentation (CT) and ACDC Cardiac MRI datasets, consistently outperforming state-of-the-art (SOTA) models like TransUNet and SynergyNet.

Qualitative comparison of L2GNet against state-of-the-art methods on Synapse and ACDC datasets.

Performance Metrics

  • Synapse Dataset: L2GNet (using 4 references) achieved a Mean Dice Similarity Coefficient (DSC) of 82.23%, significantly higher than SynergyNet (79.65%) and TransUNet (77.48%). It also drastically reduced the Hausdorff Distance (HD) from 23.59 to 14.17, indicating much more accurate boundary detection.
  • ACDC Dataset: The model achieved a 91.44% DSC, surpassing all other tested methods.

Qualitative Analysis

Visual results highlight L2GNet's ability to distinguish between adjacent organs with similar intensities, such as the liver and pancreas. While other models often suffer from inter-class "bleeding," L2GNet maintains sharp boundaries due to its structured anatomical references and transport-based global modeling.

Conclusion

L2GNet introduces a computationally efficient and anatomically grounded approach to medical image segmentation. By leveraging Optimal Transport, it captures global context without the overhead of standard self-attention. Future work includes extending the L2GMapper to native 3D volumes and exploring its integration into large-scale foundation models for medical imaging.

Summary updated

Abstract

Continuous Latent Space (CLS) and Discrete Latent Space (DLS) models, like AttnUNet and VQUNet, have excelled in medical image segmentation. In contrast, Synergistic Continuous and Discrete Latent Space (CDLS) models show promise in handling fine and coarse-grained information. However, they struggle with modeling long-range dependencies. CLS or CDLS-based models, such as TransUNet or SynergyNet are adept at capturing long-range dependencies. Since they rely heavily on feature pooling or aggregation using self-attention, they may capture dependencies among redundant regions. This hinders comprehension of anatomical structure content, poses challenges in modeling intra-class and inter-class dependencies, increases false negatives and compromises generalization. Addressing these issues, we propose L2GNet, which learns global dependencies by relating discrete codes obtained from DLS using optimal transport and aligning codes on a trainable reference. L2GNet achieves discriminative on-the-fly representation learning without an additional weight matrix in self-attention models, making it computationally efficient for medical applications. Extensive experiments on multi-organ segmentation and cardiac datasets demonstrate L2GNet's superiority over state-of-the-art methods, including the CDLS method SynergyNet, offering an novel approach to enhance deep learning models' performance in medical image analysis.
BibTeX
@misc{gorade2025l,
  author = {Vandan Gorade and Sparsh Mittal and Neethi Dasu and Rekha Singhal and KC Santosh and Debesh Jha},
  title = {{L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation}},
  year = {2025},
  eprint = {2502.05229},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.05229},
}
Figure