Introduction
Traditional robotic safety frameworks typically rely on static environmental maps and pre-calculated safety controllers. While effective for avoiding permanent structures like walls, these "baked-in" safety measures often fail to account for the dynamic and semantic nature of human-centric environments. Real-world scenarios involve transient hazards—such as a coffee spill, a fragile rug, or a cordoned-off area—that are not physical barriers but must be avoided to ensure operational safety and respect user preferences.
The challenge in addressing these semantic constraints is twofold: first, the robot must perceive and interpret high-level human intent (e.g., "Avoid the workout zone"); second, it must update its mathematically rigorous safety controllers in real-time without the prohibitive computational cost of re-calculating safety sets from scratch.
Methodology: The Safe-Lang Framework
To bridge the gap between high-level language and low-level control, we propose Safe-Lang, a pipeline that integrates Vision-Language Models (VLMs) with Hamilton-Jacobi (HJ) reachability analysis.
Perception and Semantic Mapping
When a human provides a natural language command, the system employs a pre-trained VLM (OWLv2) to identify the target objects in the robot's RGB-D camera feed. These detections are projected onto a 2D ground plane to define a "Semantic Failure Set"—a spatial representation of regions the robot must avoid.
Formal Safety via HJ Reachability
Safe-Lang utilizes HJ reachability to calculate a Backward Reachable Tube (BRT). The BRT identifies all states from which a robot is guaranteed to eventually enter the failure set, regardless of its control inputs. This provides a formal "safety shield" that monitors the robot's state relative to "doomed" configurations.
Efficient Online Updates
Calculating BRTs is traditionally computationally intensive. Safe-Lang overcomes this by using a warm-starting technique. Instead of re-calculating safety values from a blank slate, the system initializes the HJ reachability computation with previously known safety values. Since semantic updates are typically local, this allows the safety representation to be updated in approximately 3 seconds, enabling online deployment.
Experimental Evaluation
The framework was evaluated in both high-fidelity simulations and real-world hardware deployments.
- Simulation: Using the Habitat 3.0 simulator with the HSSD-HAB dataset, the robot was tasked with navigating "Home Gym" and "Hallway" environments while respecting constraints like "avoid the expensive rug."
- Hardware: A LoCoBot mobile manipulator equipped with an Intel RealSense camera was used to validate the system's ability to avoid real-world hazards like caution tape and spills based on live feedback.
Results and Discussion
Experimental results demonstrate that Safe-Lang is the only method that consistently avoids both physical obstacles and newly defined semantic constraints.
Key findings include:
- Robust Safety Filtering: The safety filter acts as a policy-agnostic shield, intervening only when the robot's nominal planner (e.g., MPPI) attempts to enter a "doomed" state.
- Language Flexibility: VLMs effectively map subjective human labels to physical coordinates, allowing the robot to respect constraints that are invisible to traditional sensors like LiDAR.
- Computational Viability: The warm-starting approach reduced update times to a level suitable for real-time interaction, maintaining safety even when commands were issued shortly before a potential collision.
Conclusion and Future Work
Safe-Lang provides a robust mechanism for updating robot safety representations online via natural language. While the current implementation uses rectangular bounding boxes, future iterations could leverage pixel-level semantic segmentation for more precise boundaries. Additionally, incorporating uncertainty-aware perception and handling dynamic semantic constraints (e.g., moving pets) remain promising avenues for future research.
Summary updated