RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning

Huiyang Hu, Peijin Wang, Yingchao Feng, Kun Wei, Wenxin Yin, Wenhui Diao, Mengyu Wang, Hanbo Bi, Kisung Kang, Ling Tong, Kun Fu, Xian Sun

Abstract

Remote sensing (RS) images from multiple modalities and platforms exhibit diverse details due to differences in sensor characteristics and imaging perspectives. Existing vision-language research in RS largely relies on relatively homogeneous data sources and remains limited to conventional visual perception tasks such as classification or captioning. To address these issues, we propose RingMo-Agent, a model designed to handle multi-modal and multi-platform data that performs perception and reasoning tasks based on user textual instructions. Supported by the large-scale RS-VL3M dataset (3 million image-text pairs), RingMo-Agent incorporates modality-adaptive representations and a unified task modeling mechanism. Extensive experiments demonstrate its effectiveness in both visual understanding and sophisticated analytical tasks, exhibiting strong generalizability across different platforms and sensing modalities.

Introduction

Remote sensing imagery presents unique challenges compared to natural images. Data is often collected from vastly different perspectives—ranging from high-altitude satellites to low-flying Unmanned Aerial Vehicles (UAVs)—and through diverse sensors including optical RGB, Synthetic Aperture Radar (SAR), and Infrared (IR). Most existing vision-language models (RSVLMs) are "single-source" and "passive," excelling at basic perception tasks like image captioning or classification but failing when faced with heterogeneous data or complex reasoning requirements.

Comparison between existing RSVLMs and the proposed RingMo-Agent framework, highlighting the expansion in platforms, modalities, and supported tasks.

RingMo-Agent bridges this gap by serving as a unified, standalone framework capable of handling multi-modal inputs and performing "agentic" reasoning. For instance, while a standard model might describe a map, RingMo-Agent can formulate a detailed flight plan for a drone, generating precise 3D waypoints and orientation angles based on textual instructions.

Methodology

RingMo-Agent utilizes a sophisticated multi-stage architecture designed for versatility and depth:

  • Modality-Aware Visual Encoder: To prevent "cross-modal interference," the model employs parallel embedding layers for Optical, SAR, and IR data. This isolation ensures the unique physical characteristics of radar or infrared sensors do not degrade the features learned for optical imagery.
  • DeepSeekMoE Backbone: The system leverages a 3-billion parameter Mixture-of-Experts (MoE) architecture, providing high reasoning capacity with computational efficiency.
  • High-Dimensional Trajectory Decoder: For spatial tasks like navigation, a specialized GRU-based decoder translates hidden states from the LLM into precise 3D coordinates.
  • Unified Task Modeling: By introducing task-specific tokens (e.g., <|navigation|>), the model shifts its internal logic to suit the required output, whether it be a simple classification or a multi-step action plan.

The RS-VL3M Dataset

A cornerstone of this research is the introduction of RS-VL3M, the first large-scale RS vision-language dataset integrating multi-platform and multi-modal data. It comprises over 3 million image-text pairs across eight distinct task types. To ensure high-quality training data, the authors used LLMs to transform sparse RS labels into rich, descriptive instructions, significantly enhancing the model's ability to understand nuance.

Experimental Results

RingMo-Agent consistently outperforms both generalist models and existing RS-specific frameworks across various benchmarks:

  • Reasoning Capability: On the ReCon1M-REL relation reasoning dataset, RingMo-Agent achieved an F1-score of 90.23%, significantly outperforming GPT-4o's 3.04%.
  • Non-Optical Detection: In IR and SAR object detection, the model achieved mAP@50 scores of 59.88% and 53.84%, respectively. Most generalist VLMs fail these tasks as they cannot interpret non-optical sensor data.
  • Autonomous Navigation: In the CityNav benchmark, RingMo-Agent approached the performance of specialized navigation models despite being a general-purpose agent.
An example of RingMo-Agent performing a complex task scheduling instruction, generating a trajectory for a quadcopter based on landmark coordinates.

Conclusion

RingMo-Agent represents a significant advancement in remote sensing, moving the field from passive perception toward active, multi-modal reasoning. While challenges remain regarding real-time deployment on low-power UAV hardware and deepening logical integration for disaster scenarios, the model establishes a new state-of-the-art for unified RS foundation models. Future work will focus on "closed-loop" systems that incorporate real-time sensor feedback for dynamic plan adjustment.

Summary updated

Abstract

Remote sensing (RS) images from multiple modalities and platforms exhibit diverse details due to differences in sensor characteristics and imaging perspectives. Existing vision-language research in RS largely relies on relatively homogeneous data sources. Moreover, they still remain limited to conventional visual perception tasks such as classification or captioning. As a result, these methods fail to serve as a unified and standalone framework capable of effectively handling RS imagery from diverse sources in real-world applications. To address these issues, we propose RingMo-Agent, a model designed to handle multi-modal and multi-platform data that performs perception and reasoning tasks based on user textual instructions. Compared with existing models, RingMo-Agent 1) is supported by a large-scale vision-language dataset named RS-VL3M, comprising over 3 million image-text pairs, spanning optical, SAR, and infrared (IR) modalities collected from both satellite and UAV platforms, covering perception and challenging reasoning tasks; 2) learns modality adaptive representations by incorporating separated embedding layers to construct isolated features for heterogeneous modalities and reduce cross-modal interference; 3) unifies task modeling by introducing task-specific tokens and employing a token-based high-dimensional hidden state decoding mechanism designed for long-horizon spatial tasks. Extensive experiments on various RS vision-language tasks demonstrate that RingMo-Agent not only proves effective in both visual understanding and sophisticated analytical tasks, but also exhibits strong generalizability across different platforms and sensing modalities.
BibTeX
@misc{hu2025ringmo,
  author = {Huiyang Hu and Peijin Wang and Yingchao Feng and Kun Wei and Wenxin Yin and Wenhui Diao and Mengyu Wang and Hanbo Bi and Kisung Kang and Ling Tong and Kun Fu and Xian Sun},
  title = {{RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning}},
  year = {2025},
  eprint = {2507.20776},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.20776},
}
Figure