RescueVLM: Model-Agnostic Vision-Language Framework for Rescue Drone Imagery

Niklas Voigt, Tim Sibum, Julien Meine, Niklas Digakis, Hartmut SurmannarXiv 2026 · arXiv

TL;DR

RescueVLM is an open-source, on-premise triage tool that uses local Vision-Language Models (VLMs) to help rescue commanders quickly find critical information in hundreds of drone images. By treating the VLM as a swappable engine and using natural language prompts instead of fine-tuning, it bypasses strict privacy restrictions and the "maintenance treadmill" of constantly retraining specialized detectors.

What Problem Does This Paper Address?

In disaster response, drones are a double-edged sword. While they provide vital situational awareness, a single mission can generate hundreds of high-resolution images. Rescue commanders, operating under extreme time pressure, cannot manually review every frame. Furthermore, constant exposure to imagery of destruction and victims carries a documented psychological toll on human operators.

Existing AI solutions often fall short in Search and Rescue (SAR) for two reasons:

  1. Privacy and Regulation: Under GDPR and German civil protection rules, sensitive disaster imagery (containing victims or private property) cannot be uploaded to cloud-based APIs like GPT-4.
  2. The "Fine-Tuning Treadmill": Specialized detectors for specific hazards (like "collapsed roofs" or "chemical spills") age quickly. SAR organizations rarely have the resources to constantly retrain and re-certify models as new datasets or better architectures emerge.

Methods

RescueVLM is built on SketchVLM, a framework that allows VLMs to "explain" their reasoning by emitting editable SVG overlays on top of images.

Key technical features include:

  • Model-Agnostic Architecture: The system uses a local Ollama backend. This allows users to swap the underlying "engine" (e.g., switching from a smaller model to a larger one like gemma:27b) via a simple dropdown without changing any code.
  • Training-Free Triage: Instead of training on SAR data, all task knowledge is embedded in a system prompt and an output contract. The model is instructed to return structured JSON containing a situation summary, a ranked list of relevant images, and SVG coordinates for visual grounding.
  • Bulk Processing & Deduplication: The tool processes images in batches (up to 50). Because drone flights often capture the same object from multiple overlapping angles, the system includes explicit instructions for the model to deduplicate events, ensuring the commander sees a unique "shortlist" rather than five photos of the same hole in a roof.
  • Geospatial Integration: It extracts EXIF-GPS data client-side to plot identified "hits" on a map, providing immediate spatial context.

Datasets

The authors evaluated RescueVLM using four raw, unlabeled image folders from real-world deployments and exercises conducted by the German Rescue Robotics Center (DRZ):

  1. ABC Reconnaissance: Post-fire survey of a burned-out industrial hall in Berlin (167 images).
  2. Flood: Imagery from the 2021 Erftstadt/Blessem flood disaster (140 images).
  3. Structural Fire: Reconnaissance of a collapsed apartment building in Essen (159 images).
  4. Vegetation Fire: A live fire exercise in Viersen (52 images).

Key Insights and Ideas

  • The Prompt is the Program: By moving task logic from "weights" to "prompts," the framework becomes durable. If a better open-source model is released tomorrow, RescueVLM improves instantly without retraining.
  • Inverting the Workflow: Instead of a human looking at 100% of the images to find the 5% that matter, the AI surfaces the 5% first. This significantly reduces the "psychological cost" of screening traumatic imagery.
  • Appropriate Reliance: The authors emphasize that the tool is a triage aid, not a diagnostic one. By providing editable annotations and keeping the full image set "one click away," they encourage commanders to verify the AI's suggestions rather than blindly trusting them.
  • Local is a Requirement, Not an Option: The paper argues that for SAR, on-premise inference isn't just a performance optimization—it is the only legal way to deploy VLMs in the field.

Key Results

  • Significant Exposure Reduction: In the vegetation fire use case, the system condensed a 52-image mission into a 5-image shortlist—a ~90% reduction in the volume of imagery a commander needed to review initially.
  • Zero-Shot Versatility: Without specific training, the model successfully identified complex hazards, such as "fragmented land segments prone to collapse" in flood zones and "traffic risks near unstable structures" at fire scenes.
  • Performance: On a single high-end consumer GPU (RTX 5090), the system processed 50 images in under a minute.

Limitations and Future Work

  • Model Reliability: The system is only as good as its underlying VLM. The authors noted that some models (like nemotron3) suffered from hallucinations, confidently labeling non-existent people or fire trucks.
  • Hardware Constraints: While model-agnostic, the most powerful open-weight models (e.g., Qwen 397B) require more VRAM than currently available on standard responder-grade workstations.
  • Quantitative Evaluation: The current study is qualitative. The authors identify the need for formal field trials with SAR experts to measure "operational utility" beyond simple accuracy metrics.
  • Future Directions: The team plans to add "action recommendations" (suggesting what the commander should do next) and support for live drone video streams.

Summary updated

Abstract

A single unmanned aerial vehicle (UAV) mission over an incident site produces hundreds of images, far more than a rescue commander can review under time pressure, and human screening of destruction imagery carries a documented psychological cost. The two obvious remedies both fail in search and rescue (SAR) practice: cloud vision-language model (VLM) services are excluded because the imagery must not leave the incident network under GDPR and German civil-protection (BOS) rules, and fine-tuned specialist detectors age quickly while retraining and re-certifying them exceeds most SAR organizations resources. We present RescueVLM, an open-source, training-free and model-agnostic triage tool built on SketchVLM: a general-purpose open-weight VLM runs fully on-premise and is swapped for a newer model through a simple dropdown selection, so triage quality grows with the underlying models because all task knowledge lives in prompts and an output contract rather than in trained weights. One natural-language query over a mission folder returns a ranked shortlist with per-image justifications, editable SVG overlay annotations, and a GPS map of the hits. In a qualitative evaluation on German fire-brigade field data from three real deployments and a live fire exercise, a single query condenses an entire survey mission into a handful of annotated frames that a commander can review at a glance.
BibTeX
@misc{voigt2026rescuevlm,
  author = {Niklas Voigt and Tim Sibum and Julien Meine and Niklas Digakis and Hartmut Surmann},
  title = {{RescueVLM: Model-Agnostic Vision-Language Framework for Rescue Drone Imagery}},
  year = {2026},
  eprint = {2611.18813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18813},
}
Figure