Abstract
In the rapidly evolving field of vision-language navigation (VLN), ensuring safety for physical agents remains an open challenge. For a human-in-the-loop language-operated drone to navigate safely, it must understand natural language commands, perceive the environment, and simultaneously avoid hazards in real time. Control Barrier Functions (CBFs) are formal methods that enforce safe operating conditions. Model Predictive Control (MPC) is an optimization framework that plans a sequence of future actions over a prediction horizon, ensuring smooth trajectory tracking while obeying constraints. In this work, we consider a VLN-operated drone platform and enhance its safety by formulating a novel scene-aware CBF that leverages ego-centric observations from a camera which has both Red-Green-Blue as well as Depth (RGB-D) channels. A CBF-less baseline system uses a Vision-Language Encoder with cross-modal attention to convert commands into an ordered sequence of landmarks. An object detection model identifies and verifies these landmarks in the captured images to generate a planned path. To further enhance safety, an Adaptive Safety Margin Algorithm (ASMA) is proposed. ASMA tracks moving objects and performs scene-aware CBF evaluation on-the-fly, which serves as an additional constraint within the MPC framework. By continuously identifying potentially risky observations, the system performs prediction in real time about unsafe conditions and proactively adjusts its control actions to maintain safe navigation throughout the trajectory. Deployed on a Parrot Bebop2 quadrotor in the Gazebo environment using the Robot Operating System (ROS), ASMA achieves 64%-67% increase in success rates with only a slight increase (1.4%-5.8%) in trajectory lengths compared to the baseline CBF-less VLN.
Introduction
Vision-Language Navigation (VLN) allows robots to follow complex natural language commands like "Go past the traffic light and land in front of the blue car." While high-level semantic understanding has improved significantly, low-level safety remains a critical bottleneck. In dynamic environments, a drone might correctly identify its target but collide with a moving pedestrian or bird because its navigation controller lacks formal safety guarantees. ASMA addresses this by bridging the gap between semantic context and control theory.
Methodology
System Architecture
The ASMA framework utilizes a modular, multi-threaded architecture that integrates high-level reasoning with rigorous control. The process begins with Vision-Language Grounding, where YOLOv5 performs object detection and CLIP (Contrastive Language-Image Pre-training) matches these detections to landmarks mentioned in the command. The system distinguishes between static and dynamic objects by tracking bounding box centers over time.
Scene-Aware Control Barrier Functions (CBFs)
To ensure safety, ASMA formulates a novel CBF that defines a "safe set" $C$ for the robot. Unlike standard reactive methods, these CBFs use real-time RGB-D data to calculate safety margins based on obstacle distance and the relative angle between the drone's target and the hazard.
MPC Integration and Adaptive Fallback
Safety constraints are embedded directly into a Model Predictive Control (MPC) framework. The drone simulates its future trajectory over a short horizon, selecting the optimal path that satisfies both the mission goal and the CBF safety constraints. If the primary detector fails to find a landmark, an adaptive fallback triggers a region proposal strategy using CLIP to ensure the mission continues without getting "stuck."
Experimental Results
The system was evaluated in custom ROS-Gazebo simulation environments, including a complex "city" environment with narrow alleys and dynamic agents.
- Safety and Success: ASMA achieved a 64%–67% increase in success rates compared to a baseline VLN system without CBF safety layers.
- Efficiency: The cost of safety was remarkably low; safety-aware trajectories were only 1.4%–5.8% longer than the baseline paths.
- Grounding Accuracy: The vision-language encoder demonstrated 92%–98% accuracy in identifying landmarks in a zero-shot setting.
- Real-time Performance: The system maintained a control frequency of 5 Hz, sufficient for real-time navigation on modern hardware.
Conclusion and Future Work
ASMA demonstrates that proactive safety planning via CBFs can significantly improve VLN reliability without sacrificing efficiency. Future research will focus on incorporating visual memory to handle occlusions and probabilistic CBFs to better manage sensor noise and uncertainty in dynamic environments.
Summary updated