RSRWKV: A Linear-Complexity 2D Attention Mechanism for Efficient Remote Sensing Vision Task

Chunshan Li, Rong Wang, Xiaofei Yang, Dianhui ChuIEEE TCSVT 2025 · DOI · OpenReview

TL;DR

This paper introduces RSRWKV, a novel vision architecture for remote sensing that addresses the computational limitations of Vision Transformers (ViTs) while improving upon existing linear-complexity models. By proposing a 2D-WKV attention mechanism and an MVC-Shift module, RSRWKV achieves efficient, isotropic spatial reasoning, leading to state-of-the-art performance in classification, detection, and segmentation tasks on high-resolution remote sensing imagery.

What Problem Does This Paper Address?

High-resolution remote sensing imagery presents a significant challenge for computer vision models due to its complexity and scale diversity. Traditional Convolutional Neural Networks (CNNs) excel at capturing local features but struggle with long-range dependencies due to their fixed receptive fields. Vision Transformers (ViTs) can model global relationships effectively but suffer from quadratic computational complexity, making them impractical for large, high-resolution images. While recent advancements like the RWKV model offer linear complexity for sequential data (like text), their adaptation to 2D vision tasks has been limited by anisotropic 1D scanning mechanisms, hindering their ability to capture spatial information comprehensively. This paper aims to bridge this gap by developing an efficient, isotropic 2D attention mechanism for remote sensing tasks.

Methods

The core of RSRWKV lies in its novel 2D-WKV scanning mechanism. This mechanism extends the linear-complexity RWKV architecture to handle 2D spatial data by processing information across four principal directions simultaneously: horizontal, vertical, and two diagonals. This allows for isotropic context aggregation without requiring multiple passes, overcoming the anisotropy issues of previous 1D-based RWKV adaptations.

Key components of the RSRWKV architecture include:

  • 2D-RWKV Block: This is the fundamental building block, integrating spatial and channel mixing modules.
  • Spatial Mix Module: This module is responsible for capturing long-range spatial dependencies. It utilizes:
  • MVC-Shift (Multi-view Context Token Shift): This novel module enhances receptive field coverage by combining depthwise separable convolutions with varying dilation rates and 1x1 convolutions. It aims to capture multi-scale information and inter-channel dependencies more effectively than previous token shifting methods.
  • 2D-WKV Attention: This is the proposed attention mechanism that processes the input in four directions, enabling isotropic spatial dependency capture.
  • Channel Mix Module: This module focuses on feature fusion along the channel dimension. It incorporates:
  • ECA (Efficient Channel Attention) Module: This module strengthens cross-channel feature interaction and prioritizes semantically significant features by adaptively adjusting channel weights.

The overall architecture is a stacked arrangement of these 2D-RWKV blocks, similar to the structure of ViTs.

Datasets

The paper evaluates RSRWKV on three benchmark remote sensing datasets:

  • NWPU RESISC45: Used for image classification.
  • NWPU VHR-10.v2: Used for object detection.
  • GLH-Water: Used for semantic segmentation.

Key Insights and Ideas

  • Isotropic 2D Attention for Remote Sensing: The core insight is that for remote sensing, capturing spatial dependencies isotropically (equally in all directions) is crucial. The 2D-WKV mechanism directly addresses this by processing information in multiple directions simultaneously, unlike previous 1D-scanning approaches.
  • Bridging Linear Complexity and 2D Reasoning: RSRWKV successfully combines the linear complexity benefits of RWKV with the need for 2D spatial reasoning in vision tasks, offering a scalable solution for high-resolution imagery.
  • MVC-Shift for Enhanced Context: The MVC-Shift module is a key innovation for improving local context capture by integrating multi-scale convolutions and inter-channel interactions, leading to better feature representation.
  • ECA for Semantic Saliency: The integration of ECA modules effectively highlights important features and improves semantic understanding by focusing on relevant channels.
  • Efficiency without Sacrificing Performance: The architecture demonstrates that achieving state-of-the-art results in complex vision tasks doesn't necessarily require quadratic complexity, making it suitable for resource-constrained environments.

Key Results

RSRWKV consistently outperforms existing CNN and Transformer-based baselines across various remote sensing tasks:

  • Image Classification (NWPU RESISC45): Achieved a top-1 accuracy of 96.19% and an F1 score of 96.18%, surpassing methods like StarNet S4, VRWKV-T, LWGANet L2, ResNet50, and Twins-PCPVT, while maintaining significantly lower computational costs.
  • Object Detection (NWPU VHR-10.v2): Achieved the highest AP6 of 42.5 and AP@506 of 78.3, outperforming ResNet50, ViT, and VRWKV-T, with a more efficient attention design.
  • Semantic Segmentation (GLH-Water): Achieved the highest IoU of 85.06% on water regions, outperforming PCL and VRWKV-T, and demonstrating superior detail preservation and boundary localization.

The paper also highlights the computational efficiency, with RSRWKV requiring 9.78M parameters and 17.63G FLOPs for a 512x512 input, which is competitive with other advanced models.

Limitations and Future Work

  • Limitations:
    • While RSRWKV achieves superior performance, its parameter count and FLOPs are slightly higher than some previous RWKV variants (e.g., VRWKV-T), though this is justified by the performance gains.
  • The paper acknowledges that the MVC-Shift module is inspired by previous work and further optimization might be possible.
  • Future Work:
    • Scaling Up: Exploring larger parameter configurations to fully leverage the RSRWKV architecture's potential.
  • Temporal Prediction: Integrating temporal prediction capabilities to handle dynamic remote sensing scenarios.
  • Multimodal Fusion: Extending the framework to incorporate multimodal data for more comprehensive scene understanding.

Summary updated

Abstract

High-resolution remote sensing analysis faces challenges in global context modeling due to scene complexity and scale diversity. While CNNs excel at local feature extraction via parameter sharing, their fixed receptive fields fundamentally restrict long-range dependency modeling. Vision Transformers (ViTs) effectively capture global semantic relationships through self-attention mechanisms but suffer from quadratic computational complexity relative to image resolution, creating critical efficiency bottlenecks for high-resolution imagery. The RWKV model's linear-complexity sequence modeling achieves breakthroughs in NLP but exhibits anisotropic limitations in vision tasks due to its 1D scanning mechanism. To address these challenges, we propose RSRWKV, featuring a novel 2D-WKV scanning mechanism that bridges sequential processing and 2D spatial reasoning while maintaining linear complexity. This enables isotropic context aggregation across multiple directions. The MVC-Shift module enhances multi-scale receptive field coverage, while the ECA module strengthens cross-channel feature interaction and semantic saliency modeling. Experimental results demonstrate RSRWKV's superior performance over CNN and Transformer baselines in classification, detection, and segmentation tasks on NWPU RESISC45, VHR-10.v2, and GLH-Water datasets, offering a scalable solution for high-resolution remote sensing analysis.
BibTeX
@article{li2025rsrwkv,
  author = {Chunshan Li and Rong Wang and Xiaofei Yang and Dianhui Chu},
  title = {{RSRWKV: A Linear-Complexity 2D Attention Mechanism for Efficient Remote Sensing Vision Task}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  doi = {10.1109/tcsvt.2025.3636726},
}
Figure