Abstract
SafeHumanoid is a proactive safety framework for humanoid robots that leverages Vision-Language Models (VLMs) and Retrieval-Augmented Generation (RAG) to dynamically adjust impedance parameters based on environmental context. By interpreting visual cues such as human proximity and object fragility from an egocentric camera, the system transitions between rigid and compliant control states. This approach ensures safe human-robot collaboration by pre-setting physical behaviors before potential collisions occur, bridging the gap between high-level semantic reasoning and low-level motor control.
Introduction
In the field of Human-Robot Interaction (HRI), safety mechanisms have traditionally been categorized as either geometric (maintaining exclusion zones) or reactive (stopping after a collision is detected). However, these methods often leave robots "semantically blind." A robot may detect a person's presence but fail to distinguish between a passive observer and an active collaborator. Consequently, robots typically operate with fixed, rigid control settings that are either hazardous in close quarters or too conservative for practical use.
SafeHumanoid addresses this by enabling robots to use "common sense" reasoning—such as recognizing the fragility of a glass or the intent of a human hand—to modulate joint stiffness and movement speed proactively.
Methods
The SafeHumanoid pipeline is a modular system distributed between an onboard controller and an offboard reasoning server. The architecture is designed to provide high-level context awareness while maintaining stable low-level control.
- Egocentric Perception (VLM): The robot’s head camera (e.g., Intel RealSense D435i) streams frames to an offboard server running Molmo-7B. The VLM processes these images using structured prompts to identify task types, object fragility, and human presence.
- Semantic-to-Safety Mapping (RAG): To mitigate the risk of AI hallucinations, the system employs Retrieval-Augmented Generation. The VLM’s textual description is converted into a vector embedding and matched against a Curated Scenario Database using FAISS. This ensures the robot only executes control parameters ($K_p$, $K_d$, and velocity $v$) that have been pre-validated for safety.
- Impedance Control: The retrieved parameters are transmitted back to the robot's onboard PC, which executes a 50Hz joint-space impedance controller.
- Fallback Logic: In cases of high uncertainty or weak RAG matches, the system defaults to a conservative, high-compliance "safe" profile.
Datasets and Validation
The researchers developed a Curated Scenario Database containing 16 validated scenarios specifically for the Unitree G1 humanoid. These scenarios cover common tabletop tasks such as picking cubes, pouring liquids, and human handovers. Each entry pairs a natural language description with a payload of 28 joint-level impedance parameters. Crucially, every parameter set was cross-checked against ISO/TS 15066 international safety standards to ensure that potential contact forces remain below human pain thresholds.
Key Results
The system was evaluated on a Unitree G1 humanoid across various tabletop manipulation tasks. The results demonstrate that the VLM-RAG pipeline effectively modulates robot behavior based on real-time visual context.
- Context-Aware Adaptation: In a surface-wiping task, the robot maintained high stiffness for effective cleaning until a human hand entered the workspace. The system immediately lowered $K_p$ and velocity $v$ to ensure a safe, compliant interaction.
- Generalization: The system showed an ability to handle Out-of-Distribution (OOD) objects. When presented with a "pin" (not in the database), the VLM categorized it as a small, fragile object, allowing the RAG system to retrieve a safe handover profile designed for similar items.
- Performance and Latency: While the pipeline successfully modulated impedance in all test cases, the offboard VLM-RAG loop introduced a latency of approximately 1.4 seconds. This delay represents the primary bottleneck for applying the framework to highly dynamic environments.
Discussion and Future Work
SafeHumanoid demonstrates the potential of combining semantic reasoning with physical control for safer HRI. By using RAG as a safety guardrail, the system maintains human-in-the-loop reliability while benefiting from the flexibility of large-scale VLMs.
Future research will focus on reducing inference latency through model distillation and edge deployment to enable reflex-like safety responses. Additionally, scaling the scenario database through simulation-to-real transfer and integrating depth data with semantic labels could further refine the robot's ability to adjust its physical compliance based on precise spatial relationships.
Summary updated