TL;DR
OnFly is a fully onboard, real-time framework that allows drones to follow complex natural-language instructions in new environments without prior training. By decoupling navigation decisions from progress monitoring and using a "prefix-stable" hybrid memory, it achieves a massive jump in success rates (from 26.4% to 67.8%) while maintaining flight safety.
What Problem Does This Paper Address?
Aerial Vision-Language Navigation (AVLN) is the task of getting a drone to follow instructions like, "Go along the corridor until you see a yellow trash bin, then turn right and stop at the gray sofa."
Doing this in a "zero-shot" manner—meaning the system hasn't been specifically trained on the environment it's currently in—is incredibly difficult. Existing methods usually suffer from three main issues:
- Decision Instability: They often use a single Vision-Language Model (VLM) stream to handle both high-frequency flight control and low-frequency task monitoring, leading to "thinking" delays that cause jerky, "stop-and-go" flight.
- Memory Loss: Most systems use a "sliding window" memory. As the drone moves, it forgets the beginning of the instruction or the starting point of the trajectory, leading to failures in long-distance tasks.
- The Safety-Efficiency Trade-off: Drones either fly too cautiously (slow) or too aggressively (crashing into things because the VLM's predicted "goal" was actually inside a wall).
Methods
The authors introduce OnFly, which tackles these bottlenecks through three core innovations:
- Shared-Perception Dual-Agent Architecture: Instead of one overworked brain, OnFly uses two. A Decision Agent runs at high frequency to generate immediate flight targets, while a Monitoring Agent runs at a lower frequency to track overall progress and signal when a sub-task is complete. They share a Vision Transformer (ViT) backbone to save computation but maintain separate "KV-caches" (the VLM's short-term memory) to stay efficient.
- Hybrid Keyframe-Recent-Frame Memory: To solve the "forgetting" problem, OnFly maintains a memory consisting of the first frame, a set of representative "keyframes" of the journey, and the most recent frame. Crucially, they use Prefix-Stable Serialization, which ensures the VLM doesn't have to re-process the entire history every time a new frame is added, significantly lowering latency.
- Semantic-Geometric Verifier & Planner: When the VLM suggests a goal, the system doesn't just blindly follow it. A verifier checks if the goal is semantically consistent (using VLM features) and geometrically safe (using depth data). A receding-horizon planner then generates a smooth, collision-free path to that goal.
Datasets
The researchers evaluated OnFly using:
- Simulation: 10 high-fidelity Unreal Engine scenes (factories, villas, urban blocks) via AirSim. They created a benchmark of 150 tasks categorized into object navigation, precise navigation, and long-range navigation.
- Real-World: They deployed the system on a custom quadrotor equipped with a NVIDIA Jetson Orin NX, a Mid360 LiDAR, and a Realsense D435 camera for live testing in indoor and outdoor environments.
Key Insights and Ideas
- Decoupling is Essential: By separating "where do I go right now?" from "how much of the total job is done?", the system avoids the computational lag that plagues single-stream models.
- KV-Cache Stability Matters: In the world of Large Language Models (LLMs) and VLMs, re-computing tokens is expensive. By designing a memory that keeps the "prefix" (the start of the sequence) stable, OnFly achieves a 6.5x speedup in monitoring latency compared to standard implementations.
- Geometry Refines Semantics: VLMs are great at understanding "yellow trash bin" but terrible at knowing exactly how many meters away it is. Using depth cues to "snap" the VLM's 2D pixel prediction to a 3D coordinate makes the system viable for actual flight.
Key Results
- Success Rate (SR): In simulations, OnFly achieved a 67.8% success rate, more than doubling the performance of the previous state-of-the-art baseline (SPF at 26.4%).
- Safety: The collision rate dropped to a mere 2.7%, compared to over 40% for some baselines.
- Efficiency: Flight time was reduced significantly (27.1s vs 39.2s for the nearest competitor) because the drone didn't have to stop constantly to "think."
- Onboard Feasibility: The system proved it could run entirely on edge hardware (Jetson Orin NX), proving that high-level VLM reasoning doesn't require a beefy ground station or cloud connection.
Limitations and Future Work
While OnFly is a major step forward, the authors note a few areas for growth:
- Open-World Complexity: While zero-shot, the system still relies on the underlying VLM's knowledge. Extremely niche or cluttered environments might still confuse the model.
- Inference Speed: Even with optimizations, running large VLMs on edge hardware is demanding. Future work will look into further model quantization and optimization to make the drone even more responsive.
- Longer Horizons: The current memory handles "long-range" tasks better than before, but truly massive, multi-kilometer missions will likely require even more sophisticated hierarchical memory management.
Summary updated