LeViT-UNet++: Towards Balanced Fine-Grained Details and Context for Efficient Medical Image Segmentation

Guoping Xu, Jinling Zhu, Liuyue LiarXiv 2026 · arXiv

TL;DR

LeViT-UNet++ is a lightweight hybrid architecture that bridges the gap between convolutional and Transformer-based medical image segmentation. By integrating wavelet-based downsampling and an efficient depthwise-convolution decoder, it achieves competitive accuracy with a fraction of the parameters (only ~8M) and significantly higher inference speeds (up to 241 FPS).

What Problem Does This Paper Address?

Medical image segmentation is a high-stakes task where accuracy and speed are often at odds. Clinical applications like surgical navigation and real-time image-guided intervention require models that are both precise—capturing fine-grained anatomical boundaries—and efficient enough to run on standard hardware.

Current paradigms have distinct weaknesses:

  • Convolutional Neural Networks (CNNs): Excellent at capturing local spatial details but struggle with long-range global context.
  • Transformers: Superb at modeling global dependencies but are computationally expensive and often lose fine-grained details during the aggressive downsampling required to make them efficient.

The authors identify two specific bottlenecks in existing hybrid models like LeViT-UNet: conventional downsampling (like max pooling) inevitably discards high-frequency structural information, and standard decoders are often computationally redundant, using heavy convolutions to fuse features.

Methods

LeViT-UNet++ introduces two primary architectural innovations to solve these issues:

  1. Haar Wavelet Multi-Scale Downsampling (HWMD): Instead of simple pooling, this module uses the Haar wavelet transform to decompose feature maps into low-frequency (coarse structures) and high-frequency (edges and fine details) components. By processing these components with parallel depthwise convolutions at multiple scales, the model preserves structural cues that are usually lost during spatial reduction.

    Illustration of the HWMD module
  2. Depthwise-based Local and Global feature fusion and learning decoder (DLGD): The authors replaced standard, heavy convolutional blocks in the decoder with lightweight depthwise separable convolutions. This module progressively integrates global contextual representations from the Transformer blocks with local spatial features from the encoder's skip connections. This design reduces the decoder's parameter count by nearly 50% compared to the original LeViT-UNet.

The overall architecture follows a U-shaped encoder-decoder structure, using a hybrid encoder (CNN stem followed by Transformer blocks) to balance local and global feature extraction.

Overview of the LeViT-UNet++ architecture

Datasets

The model was evaluated across four diverse benchmarks covering both 2D and 3D segmentation tasks:

  • Synapse (Abdominal CT): Multi-organ segmentation of 8 abdominal organs.
  • ACDC (Cardiac MRI): Segmentation of the left ventricle, myocardium, and right ventricle.
  • LA2018 (Left Atrium MRI): A 3D segmentation challenge for the left atrium.
  • MSD Heart (Cardiac MRI): Part of the Medical Segmentation Decathlon, focused on 3D heart structure.

Key Insights and Ideas

  • Frequency Matters: The use of Haar wavelets suggests that "downsampling" shouldn't just be about reducing resolution; it should be about intelligent frequency decomposition. Keeping high-frequency information explicitly helps the model recover sharp boundaries later in the decoder.
  • Hybrid Efficiency: The paper reinforces the idea that Transformers are most efficient when applied to low-resolution feature maps at the "bottleneck" of a network, while CNNs should handle the high-resolution "stems."
  • Decoder Redundancy: A major "aha" moment is the realization that decoders in medical imaging models are often unnecessarily heavy. By switching to depthwise convolutions, the authors maintained (and even improved) accuracy while drastically cutting the computational budget.

Key Results

LeViT-UNet++ demonstrates a superior accuracy-efficiency trade-off compared to heavyweights like TransUNet and Swin-UNet:

  • Synapse Dataset: Achieved a Dice Similarity Coefficient (DSC) of 76.57% with only 8.025M parameters and a blistering inference rate of 241.72 FPS. For comparison, TransUNet has ~67M parameters and runs at ~204 FPS with lower accuracy (75.31% DSC).
  • ACDC Dataset: Attained a DSC of 88.95% and a 95th-percentile Hausdorff Distance (HD95) of 1.35, showing excellent boundary agreement.
  • 3D Performance: On the LA2018 dataset, it boosted DSC from 81.50% (original LeViT-UNet) to 84.88% while reducing parameters by 75.6%.

Qualitatively, the model produces much crisper boundaries and fewer spurious fragments than its predecessors, as seen in the 3D reconstructions.

Qualitative comparison on 3D datasets

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Fusion Strategy: The current decoder relies on simple concatenation and depthwise convolution. More sophisticated, adaptive fusion mechanisms (like attention-guided selection) might further improve how local and global features interact.
  • Dataset Diversity: While tested on four benchmarks, further validation on larger, more clinically diverse datasets is needed to prove generalizability across different pathologies and imaging hardware.
  • Knowledge Distillation: The authors suggest that using knowledge distillation to transfer features from massive, pre-trained models into this lightweight architecture could be a promising next step for even higher performance.

Summary updated

Abstract

Efficient medical image segmentation is essential for clinical applications requiring fast and accurate delineation, such as surgical navigation, radiation therapy, and real-time image-guided intervention. Despite substantial advances in convolutional neural networks and Transformers, attaining high accuracy under limited computational resources remains difficult, because reducing model complexity often sacrifices fine-grained structural details and global context modeling, yielding an unfavorable accuracy-efficiency trade-off. To address this, we propose LeViT-UNet++, an efficient encoder-decoder architecture that better balances local detail preservation and global context modeling. Specifically, we introduce a Haar Wavelet Multi-Scale Downsampling (HWMD) module that decomposes feature representations into low- and high-frequency components to extract multi-scale structural information, thereby mitigating detail loss during downsampling. We further develop a depthwise convolution-based decoder that integrates multi-scale global representations from Transformer blocks with local spatial features, enabling progressive feature refinement while reducing computational overhead. Evaluated on four benchmarks (Synapse, ACDC, LA2018, and MSD Heart) under both 2D and 3D settings, LeViT-UNet++ achieves competitive accuracy with a compact parameter footprint and fast inference. On Synapse, for instance, it attains a DSC of 76.57% with only 8.025M parameters and 241.72 FPS, confirming a favorable accuracy-efficiency balance. These results demonstrate the effectiveness of combining wavelet-based multi-scale feature extraction with lightweight local-global feature fusion for efficient medical segmentation. The source code is available at https://github.com/apple1986/levit_unet_pp.
BibTeX
@misc{xu2026levit,
  author = {Guoping Xu and Jinling Zhu and Liuyue Li},
  title = {{LeViT-UNet++: Towards Balanced Fine-Grained Details and Context for Efficient Medical Image Segmentation}},
  year = {2026},
  eprint = {2611.22482},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22482},
}
Figure