U-RWKV: Lightweight medical image segmentation with direction-adaptive RWKV

Hongbo Ye, Fenghe Tang, Peiang Zhao, Zhen Huang, Dexin Zhao, Minghao Bian, S. Kevin ZhouarXiv 2025 · arXiv

TL;DR

U-RWKV is a new lightweight medical image segmentation framework that adapts the linear-complexity ($O(N)$) Recurrent Weighted Key-Value (RWKV) architecture for 2D visual data. By combining a multi-directional scanning strategy with a stage-adaptive feature refiner, it achieves state-of-the-art segmentation accuracy while using a fraction of the parameters required by traditional Transformers and CNNs.

What Problem Does This Paper Address?

Accurate medical image segmentation is vital for clinical diagnostics, but deploying top-tier deep learning models in resource-constrained healthcare settings remains a massive challenge.

Standard Convolutional Neural Networks (CNNs), like the classic U-Net, are computationally efficient but suffer from limited Effective Receptive Fields (ERFs), meaning they struggle to capture long-range, global contextual relationships across an image. Vision Transformers (ViTs) solve this by capturing global dependencies perfectly, but they suffer from quadratic computational complexity ($O(N^2)$) relative to image size, making them too heavy for lightweight deployment.

To bridge this gap, the authors look to RWKV—an architecture that blends the parallelized training efficiency of Transformers with the linear ($O(N)$) computational scaling of Recurrent Neural Networks (RNNs). However, RWKV is inherently designed for 1D sequential data (like text). Adapting it to 2D images without introducing directional scanning bias or losing spatial context is the core challenge this paper tackles.

Methods

The authors introduce U-RWKV, a U-shaped encoder-decoder network powered by two novel components designed to optimize 2D global context processing:

  1. Direction-Adaptive RWKV Module (DARM): To process 2D images using a 1D sequence model without directional bias, DARM utilizes a QuadScan mechanism. It scans the image feature maps in four distinct directions: left-to-right, right-to-left, top-to-bottom, and bottom-to-top. These scans are processed via a Dual-RWKV spatial mixing architecture that runs forward and backward passes independently (without weight sharing), ensuring comprehensive global context aggregation.
  2. Stage-Adaptive Squeeze-and-Excitation Module (SASE): SASE dynamically alters its internal architecture based on the network depth. In shallow layers (where resolution is high), it employs dilated inverted bottlenecks to preserve fine-grained spatial details. In deeper layers, it transitions to compact bottlenecks and depthwise separable convolutions to efficiently capture high-level semantic relationships.
Fig. 2: Overview of the proposed U-RWKV; SASE Module and DARM

Datasets

The framework was rigorously evaluated across several diverse medical imaging benchmarks:

  • BUSI: Breast ultrasound images for lesion segmentation.
  • Kvasir & CVC-ClinicDB: Endoscopic video frame datasets for polyp segmentation.
  • ISIC 2017 & 2018: Skin lesion dermoscopy datasets.
  • Synapse: A multi-organ CT segmentation dataset.

Key Insights and Ideas

  • Breaking the 1D Processing Bias: Simply flattening a 2D image into a single 1D vector for an RNN-like model introduces severe directional bias. The combination of QuadScan and Dual-RWKV proves that multi-directional, independent sequence processing is key to unlocking RNN-like efficiency for vision tasks.
  • Massive Receptive Fields on a Budget: U-RWKV demonstrates that you don't need quadratic attention mechanisms ($O(N^2)$) to get a fully global receptive field. Linear attention variants can focus activation energy across the entire image just as effectively.
  • Resolution-Aware Adaptivity: Rather than using uniform blocks throughout the U-Net backbone, changing the block behavior (via SASE) based on feature map resolution balances spatial preservation and semantic abstraction perfectly.

Key Results

  • State-of-the-Art Accuracy: U-RWKV achieved the highest average Dice score of 82.27 across five distinct datasets, outperforming much heavier models like CMU-Net and TransUNet.
Comparison of mIoU vs. Parameter count across different models.
  • Extreme Parameter Efficiency: The ultra-lightweight variant, U-RWKV-s, achieved a highly competitive 80.06 average Dice score using a mere 0.46M parameters and 1.02 GFLOPs.
  • Unmatched Receptive Field: Effective Receptive Field (ERF) analysis revealed that U-RWKV achieved a 0.992 high-contribution area ratio at a 0.99 threshold, significantly outperforming the baseline U-Net (0.568) and beating out Mamba- and ViT-based alternatives (0.946).
Effective Receptive Field (ERF) comparison showing U-RWKV's superior global context capture.
Fig. 3: Visualization of segmentation results on four datasets: CVC-ClinicDB, BUSI, Kvasir-Seg, and ISIC 2017. The orange contour lines represent the ground- truth annotations of lesions, while the yellow contour lines indicate the segmen- tation results produced by our model.

Limitations and Future Work

The authors acknowledge that while U-RWKV is highly parameter-efficient, its inference speed is slightly slower than pure, highly optimized CNN-based lightweight models (such as UNeXt) due to the multi-directional scanning overhead.

Moving forward, the authors plan to focus on optimizing the framework for ultra-high-resolution imaging environments and extending the direction-adaptive RWKV mechanism from 2D slices to 3D medical image volumes.

Summary updated

Abstract

Achieving equity in healthcare accessibility requires lightweight yet high-performance solutions for medical image segmentation, particularly in resource-limited settings. Existing methods like U-Net and its variants often suffer from limited global Effective Receptive Fields (ERFs), hindering their ability to capture long-range dependencies. To address this, we propose U-RWKV, a novel framework leveraging the Recurrent Weighted Key-Value(RWKV) architecture, which achieves efficient long-range modeling at O(N) computational cost. The framework introduces two key innovations: the Direction-Adaptive RWKV Module(DARM) and the Stage-Adaptive Squeeze-and-Excitation Module(SASE). DARM employs Dual-RWKV and QuadScan mechanisms to aggregate contextual cues across images, mitigating directional bias while preserving global context and maintaining high computational efficiency. SASE dynamically adapts its architecture to different feature extraction stages, balancing high-resolution detail preservation and semantic relationship capture. Experiments demonstrate that U-RWKV achieves state-of-the-art segmentation performance with high computational efficiency, offering a practical solution for democratizing advanced medical imaging technologies in resource-constrained environments. The code is available at https://github.com/hbyecoding/U-RWKV.
BibTeX
@misc{ye2025u,
  author = {Hongbo Ye and Fenghe Tang and Peiang Zhao and Zhen Huang and Dexin Zhao and Minghao Bian and S. Kevin Zhou},
  title = {{U-RWKV: Lightweight medical image segmentation with direction-adaptive RWKV}},
  year = {2025},
  eprint = {2507.11415},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.11415},
}
Figure