OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency

Guiyong Zheng, Yueting Ban, Mingjie Zhang, Juepeng Zheng, Boyu Zhou

TL;DR

OnFly is a fully onboard, real-time framework that allows drones to follow complex natural-language instructions in new environments without prior training. By decoupling navigation decisions from progress monitoring and using a "prefix-stable" hybrid memory, it achieves a massive jump in success rates (from 26.4% to 67.8%) while maintaining flight safety.

What Problem Does This Paper Address?

Aerial Vision-Language Navigation (AVLN) is the task of getting a drone to follow instructions like, "Go along the corridor until you see a yellow trash bin, then turn right and stop at the gray sofa."

Doing this in a "zero-shot" manner—meaning the system hasn't been specifically trained on the environment it's currently in—is incredibly difficult. Existing methods usually suffer from three main issues:

  1. Decision Instability: They often use a single Vision-Language Model (VLM) stream to handle both high-frequency flight control and low-frequency task monitoring, leading to "thinking" delays that cause jerky, "stop-and-go" flight.
  2. Memory Loss: Most systems use a "sliding window" memory. As the drone moves, it forgets the beginning of the instruction or the starting point of the trajectory, leading to failures in long-distance tasks.
  3. The Safety-Efficiency Trade-off: Drones either fly too cautiously (slow) or too aggressively (crashing into things because the VLM's predicted "goal" was actually inside a wall).

Methods

The authors introduce OnFly, which tackles these bottlenecks through three core innovations:

  • Shared-Perception Dual-Agent Architecture: Instead of one overworked brain, OnFly uses two. A Decision Agent runs at high frequency to generate immediate flight targets, while a Monitoring Agent runs at a lower frequency to track overall progress and signal when a sub-task is complete. They share a Vision Transformer (ViT) backbone to save computation but maintain separate "KV-caches" (the VLM's short-term memory) to stay efficient.
  • Hybrid Keyframe-Recent-Frame Memory: To solve the "forgetting" problem, OnFly maintains a memory consisting of the first frame, a set of representative "keyframes" of the journey, and the most recent frame. Crucially, they use Prefix-Stable Serialization, which ensures the VLM doesn't have to re-process the entire history every time a new frame is added, significantly lowering latency.
  • Semantic-Geometric Verifier & Planner: When the VLM suggests a goal, the system doesn't just blindly follow it. A verifier checks if the goal is semantically consistent (using VLM features) and geometrically safe (using depth data). A receding-horizon planner then generates a smooth, collision-free path to that goal.

Datasets

The researchers evaluated OnFly using:

  • Simulation: 10 high-fidelity Unreal Engine scenes (factories, villas, urban blocks) via AirSim. They created a benchmark of 150 tasks categorized into object navigation, precise navigation, and long-range navigation.
  • Real-World: They deployed the system on a custom quadrotor equipped with a NVIDIA Jetson Orin NX, a Mid360 LiDAR, and a Realsense D435 camera for live testing in indoor and outdoor environments.

Key Insights and Ideas

  • Decoupling is Essential: By separating "where do I go right now?" from "how much of the total job is done?", the system avoids the computational lag that plagues single-stream models.
  • KV-Cache Stability Matters: In the world of Large Language Models (LLMs) and VLMs, re-computing tokens is expensive. By designing a memory that keeps the "prefix" (the start of the sequence) stable, OnFly achieves a 6.5x speedup in monitoring latency compared to standard implementations.
  • Geometry Refines Semantics: VLMs are great at understanding "yellow trash bin" but terrible at knowing exactly how many meters away it is. Using depth cues to "snap" the VLM's 2D pixel prediction to a 3D coordinate makes the system viable for actual flight.

Key Results

  • Success Rate (SR): In simulations, OnFly achieved a 67.8% success rate, more than doubling the performance of the previous state-of-the-art baseline (SPF at 26.4%).
  • Safety: The collision rate dropped to a mere 2.7%, compared to over 40% for some baselines.
  • Efficiency: Flight time was reduced significantly (27.1s vs 39.2s for the nearest competitor) because the drone didn't have to stop constantly to "think."
  • Onboard Feasibility: The system proved it could run entirely on edge hardware (Jetson Orin NX), proving that high-level VLM reasoning doesn't require a beefy ground station or cloud connection.

Limitations and Future Work

While OnFly is a major step forward, the authors note a few areas for growth:

  • Open-World Complexity: While zero-shot, the system still relies on the underlying VLM's knowledge. Extremely niche or cluttered environments might still confuse the model.
  • Inference Speed: Even with optimizations, running large VLMs on edge hardware is demanding. Future work will look into further model quantization and optimization to make the drone even more responsive.
  • Longer Horizons: The current memory handles "long-range" tasks better than before, but truly massive, multi-kilometer missions will likely require even more sophisticated hierarchical memory management.

Summary updated

Abstract

Aerial vision-language navigation (AVLN) enables UAVs to follow natural-language instructions in complex 3D environments. However, existing zero-shot AVLN methods often suffer from unstable single-stream Vision-Language Model decision-making, unreliable long-horizon progress monitoring, and a trade-off between safety and efficiency. We propose OnFly, a fully onboard, real-time framework for zero-shot AVLN. OnFly adopts a shared-perception dual-agent architecture that decouples high-frequency target generation from low-frequency progress monitoring, thereby stabilizing decision-making. It further employs a hybrid keyframe-recent-frame memory to preserve global trajectory context while maintaining KV-cache prefix stability, enabling reliable long-horizon monitoring with termination and recovery signals. In addition, a semantic-geometric verifier refines VLM-predicted targets for instruction consistency and geometric safety using VLM features and depth cues, while a receding-horizon planner generates optimized collision-free trajectories under geometric safety constraints, improving both safety and efficiency. In simulation, OnFly improves task success from 26.4% to 67.8%, compared with the strongest state-of-the-art baseline, while fully onboard real-world flights validate its feasibility for real-time deployment. The code will be released at https://github.com/Robotics-STAR-Lab/OnFly
BibTeX
@misc{zheng2026onfly,
  author = {Guiyong Zheng and Yueting Ban and Mingjie Zhang and Juepeng Zheng and Boyu Zhou},
  title = {{OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency}},
  year = {2026},
  eprint = {2603.10682},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.10682},
}

Loading related content...

Figure