LightFormer: A lightweight and efficient decoder for remote sensing image segmentation

Sihang Chen, Lijun Yun, Ze Liu, Jianfeng Zhu, Jie Chen, Hui Wang, Yueping Nie

TL;DR

This paper introduces LightFormer, a novel lightweight decoder for remote sensing image segmentation that significantly reduces computational cost while maintaining high accuracy. It's particularly effective for time-critical applications like disaster response, where processing speed and efficiency are crucial for analyzing complex scenes with unstructured targets.

What Problem Does This Paper Address?

Semantic segmentation of remote sensing images is vital for tasks like land-use classification and disaster monitoring. However, existing deep learning models, especially their decoders, are often computationally expensive, hindering their real-time deployment on resource-constrained devices like drones. This is especially problematic in scenarios with unstructured targets (e.g., damaged buildings after a disaster) where precise segmentation is needed quickly. The paper aims to bridge this gap by developing a decoder that is both efficient and accurate, even for challenging, complex scenes.

Methods

LightFormer is a CNN-Transformer hybrid decoder designed to be lightweight and efficient. Its core innovations include:

  • Lightweight Channel Refinement Module (LCRM): This module efficiently fuses features from CNN and Transformer branches. It achieves this by splitting feature channels, processing them separately in global context and local detail branches, and then recombining them. This approach drastically reduces parameters and FLOPs compared to conventional hybrid modules.
  • Cross-Scale Feature Fusion Module (CFFM): This module addresses the challenge of integrating multi-scale features from different encoder layers. It upsamples higher-level features to match lower-level ones and uses learnable weights to combine them, preventing dominance by any single scale.
  • Spatial Information Selection Module (SISM): This module is crucial for handling unstructured targets. It employs parallel pathways with large and small receptive fields to adaptively capture both global context and fine-grained local details. This allows for more precise recognition of objects in complex backgrounds.

Datasets

The authors evaluated LightFormer on several datasets:

  • LoveDA: A dataset for land-cover mapping in urban and rural areas.
  • FloodNet: A UAV imagery dataset focused on disaster scenarios, specifically flood impacts.
  • RescueNet: Another UAV dataset for disaster scenarios, focusing on building and road damage.
  • ISPRS Potsdam & ISPRS Vaihingen: Established benchmarks for urban scene semantic segmentation.

Key Insights and Ideas

  • Decoder Efficiency is Crucial: While much research focuses on lightweight encoders, the decoder's computational cost is a significant bottleneck for real-time applications. LightFormer directly tackles this by redesigning the decoder architecture.
  • Hybrid CNN-Transformer for Efficiency: By carefully managing channel splitting and using lightweight operations, the paper demonstrates that a hybrid approach can achieve strong performance without the high computational cost often associated with Transformers.
  • SISM for Unstructured Targets: The SISM module's ability to adaptively fuse multi-scale spatial information is key to its success in challenging scenarios with damaged or irregular objects.
  • Modular Design for Flexibility: The proposed modules (LCRM, CFFM, SISM) are designed to be plug-and-play, allowing for integration into various existing architectures.

Key Results

LightFormer demonstrates a strong accuracy-efficiency trade-off across multiple datasets:

  • ISPRS Vaihingen: Achieved 99.9% of GLFFNet's mIoU (83.9% vs. 84.0%) while using only 15.9% of its parameters and 14.7% of its FLOPs.
  • FloodNet: Outperformed other methods with a 1.6% improvement in mIoU over LoveNAS, using only 2.81% of its parameters and 2.45% of its FLOPs.
  • RescueNet: Showed improvements in mIoU over SFA-Net and UNetFormer, with significant reductions in FLOPs.
  • LoveDA: Consistently achieved competitive or superior performance across various backbone networks, particularly excelling in the "Barren" category.
  • ISPRS Potsdam: Delivered comparable results to SOTA methods with significantly fewer parameters and FLOPs.

Limitations and Future Work

  • Computational Cost of Attention: The authors acknowledge that while attention mechanisms are effective for long-range dependencies, computing attention for every image patch still incurs higher computational costs compared to purely CNN-based decoders.
  • Future Work: The paper suggests exploring advanced attention optimization strategies, inspired by large language models, to further alleviate computational bottlenecks. They also plan to investigate LightFormer's adaptability to multi-source data fusion and its compatibility with large-scale foundation models for remote sensing.

Summary updated

Abstract

Deep learning techniques have achieved remarkable success in the semantic segmentation of remote sensing images and in land-use change detection. Nevertheless, their real-time deployment on edge platforms remains constrained by decoder complexity. Herein, we introduce LightFormer, a lightweight decoder for time-critical tasks that involve unstructured targets, such as disaster assessment, unmanned aerial vehicle search-and-rescue, and cultural heritage monitoring. LightFormer employs a feature-fusion and refinement module built on channel processing and a learnable gating mechanism to aggregate multi-scale, multi-range information efficiently, which drastically curtails model complexity. Furthermore, we propose a spatial information selection module (SISM) that integrates long-range attention with a detail preservation branch to capture spatial dependencies across multiple scales, thereby substantially improving the recognition of unstructured targets in complex scenes. On the ISPRS Vaihingen benchmark, LightFormer attains 99.9% of GLFFNet's mIoU (83.9% vs. 84.0%) while requiring only 14.7% of its FLOPs and 15.9% of its parameters, thus achieving an excellent accuracy-efficiency trade-off. Consistent results on LoveDA, ISPRS Potsdam, RescueNet, and FloodNet further demonstrate its robustness and superior perception of unstructured objects. These findings highlight LightFormer as a practical solution for remote sensing applications where both computational economy and high-precision segmentation are imperative.
BibTeX
@misc{chen2025lightformer,
  author = {Sihang Chen and Lijun Yun and Ze Liu and Jianfeng Zhu and Jie Chen and Hui Wang and Yueping Nie},
  title = {{LightFormer: A lightweight and efficient decoder for remote sensing image segmentation}},
  year = {2025},
  eprint = {2504.10834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.10834},
}
Figure