RescueADI: Adaptive Disaster Interpretation in Remote Sensing Images with Autonomous Agents

Zhuoran Liu, Danpei Zhao, Bo Yuan

Abstract

Current methods for disaster scene interpretation in remote sensing images (RSIs) mostly focus on isolated tasks such as segmentation, detection, or visual question-answering (VQA). However, current interpretation methods often fail at tasks that require the combination of multiple perception methods and specialized tools. To fill this gap, this paper introduces Adaptive Disaster Interpretation (ADI), a novel task designed to solve requests by planning and executing multiple sequentially correlative interpretation tasks to provide a comprehensive analysis of disaster scenes. To facilitate research and application in this area, we present a new dataset named RescueADI, which contains high-resolution RSIs with annotations for three connected aspects: planning, perception, and recognition. The dataset includes 4,044 RSIs, 16,949 semantic masks, 14,483 object bounding boxes, and 13,424 interpretation requests across nine challenging request types. Moreover, we propose a new disaster interpretation method employing autonomous agents driven by large language models (LLMs) for task planning and execution, proving its efficacy in handling complex disaster interpretations. The proposed agent-based method solves various complex interpretation requests such as counting, area calculation, and path-finding without human intervention, which traditional single-task approaches cannot handle effectively. Experimental results on RescueADI demonstrate the feasibility of the proposed task and show that our method achieves an accuracy 9% higher than existing VQA methods, highlighting its advantages over conventional disaster interpretation approaches.

Introduction

When a disaster strikes, rescuers need more than just a "yes/no" answer or a highlighted map; they need actionable intelligence. Current remote sensing interpretation usually falls into two camps: isolated perception models that perform single tasks (e.g., road segmentation) without connecting data to a larger goal, and Visual Question Answering (VQA) models that answer natural language questions but often "hallucinate" numbers and lack transparency in their reasoning.

The gap lies in complex reasoning. If a rescuer asks, "Is there a clear path for an ambulance between these two points?", a model must segment the road, detect debris, and run a pathfinding algorithm. Single-task models cannot do this, and end-to-end black-box models are not reliable enough for high-stakes rescue missions.

Methods: The ADI Framework

The authors propose an agent-based framework that breaks the interpretation process into three distinct, interpretable stages: Planning, Perception, and Recognition.

Comparison of traditional visual perception, VQA, and the proposed Adaptive Disaster Interpretation (ADI) framework.
  • Planning Module: An LLM (like GPT-4o-mini) acts as the "brain." It receives a user request and generates a structured JSON plan, selecting which tools to use and in what order.
  • Perception Module: This is the "eyes" of the system. It uses state-of-the-art specialized models like GroundingDINO for object detection and PSPNet for semantic segmentation. By using dedicated tools rather than a general-purpose vision-language model, the system extracts much more accurate spatial data.
  • Recognition Module: This module processes the raw visual data into answers. It includes specialized non-neural tools for A* pathfinding, area calculation (using Ground Sample Distance), and object counting. Finally, a summarizing LLM takes these results and crafts a natural language response.

The RescueADI Dataset

The paper introduces the RescueADI dataset, a benchmark specifically designed for autonomous agents in disaster scenarios. Built upon the existing RescueNet dataset, it contains 4,044 high-resolution remote sensing images and 13,424 interpretation requests.

Comparison of different tasks in the RescueADI dataset, including object detection, semantic segmentation, VQA, and the complex ADI task.

The dataset is unique because it provides ground truth for the entire pipeline: the required plan, the perception masks/boxes, and the final textual answer. It includes nine request types, ranging from simple object existence to complex area calculations and rescue path searches.

Key Insights and Results

The primary innovation is modularity over end-to-end learning. For quantitative tasks like counting or measuring area, specialized algorithms are far superior to neural "guesses." By decoupling planning from execution, the agent avoids the common VLM pitfall of hallucinating numbers. Furthermore, transparency is a key safety feature; because the agent produces intermediate outputs (like a segmentation mask), a human operator can verify the agent's logic.

Experimental results show:

  • Superior Accuracy: The proposed ADI method achieved an accuracy 9% higher than existing state-of-the-art VQA models like GeoChat and VisualGLM.
  • Robust Planning: Using GPT-4o-mini as a backbone, the agent generated valid, executable plans in over 99% of cases, with a planning precision of 90.51%.
  • Numerical Reliability: In tasks involving counting and area calculation, the agent-based approach showed significant gains because it relied on deterministic tools rather than probabilistic token prediction.

Limitations and Future Work

There is a significant performance drop when using smaller, open-source LLMs. While GPT-4o-mini excelled, Llama-3 8B struggled with complex planning, suggesting the method currently relies on high-tier proprietary models. Additionally, while the dataset covers many scenarios, expanding the variety of disaster types is necessary for global applicability. Future work will focus on improving framework robustness and exploring "open-world" disaster requests that are not predefined in the current toolset.

Summary updated

Abstract

Current methods for disaster scene interpretation in remote sensing images (RSIs) mostly focus on isolated tasks such as segmentation, detection, or visual question-answering (VQA). However, current interpretation methods often fail at tasks that require the combination of multiple perception methods and specialized tools. To fill this gap, this paper introduces Adaptive Disaster Interpretation (ADI), a novel task designed to solve requests by planning and executing multiple sequentially correlative interpretation tasks to provide a comprehensive analysis of disaster scenes. To facilitate research and application in this area, we present a new dataset named RescueADI, which contains high-resolution RSIs with annotations for three connected aspects: planning, perception, and recognition. The dataset includes 4,044 RSIs, 16,949 semantic masks, 14,483 object bounding boxes, and 13,424 interpretation requests across nine challenging request types. Moreover, we propose a new disaster interpretation method employing autonomous agents driven by large language models (LLMs) for task planning and execution, proving its efficacy in handling complex disaster interpretations. The proposed agent-based method solves various complex interpretation requests such as counting, area calculation, and path-finding without human intervention, which traditional single-task approaches cannot handle effectively. Experimental results on RescueADI demonstrate the feasibility of the proposed task and show that our method achieves an accuracy 9% higher than existing VQA methods, highlighting its advantages over conventional disaster interpretation approaches. The dataset will be publicly available.
BibTeX
@misc{liu2024rescueadi,
  author = {Zhuoran Liu and Danpei Zhao and Bo Yuan},
  title = {{RescueADI: Adaptive Disaster Interpretation in Remote Sensing Images with Autonomous Agents}},
  year = {2024},
  eprint = {2410.13384},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.13384},
}
Figure