Abstract
Robotic guidance systems have shown promise in supporting blind and visually impaired (BVI) individuals with wayfinding and obstacle avoidance. However, most existing systems assume a clear path and do not support a critical aspect of navigation—environmental interactions that require manipulating objects to enable movement. These interactions are challenging for a human–robot pair because they demand (i) precise localization and manipulation of interaction targets (e.g., pressing elevator buttons) and (ii) dynamic coordination between the user’s and robot’s movements (e.g., pulling out a chair to sit). We present a collaborative human–robot approach that combines our robotic guide dog’s precise sensing and localization capabilities with the user’s ability to perform physical manipulation. The system alternates between two modes: lead mode, where the robot detects and guides the user to the target, and adaptation mode, where the robot adjusts its motion as the user interacts with the environment (e.g., opening a door). Evaluation results show that our system enables navigation that is safer, smoother, and more efficient than both a traditional white cane and a non-adaptive guiding system, with the performance gap widening as tasks demand higher precision in locating interaction targets. These findings highlight the promise of human–robot collaboration in advancing assistive technologies toward more generalizable and realistic navigation support.
Introduction
For the blind and visually impaired (BVI) community, navigating complex environments involves more than just moving from point A to point B. While current robotic guides are effective at wayfinding and obstacle avoidance, they often struggle with "environmental interactions"—the fine-grained tasks required to progress through a space, such as opening doors or pressing elevator buttons. These tasks present a fundamental asymmetry: robots excel at precise sensing and localization but struggle with complex physical manipulation, whereas humans are adept at manipulation but lack the visual data to find small targets in unfamiliar spaces. This research introduces a collaborative framework that bridges this gap by treating the human and robot as a coordinated team.
Methods: The Dual-Mode Collaboration Pipeline
The system is built upon a Unitree Go2 quadruped robot equipped with a 360° camera and LiDAR. The core of the interaction logic is a dual-mode framework designed to manage the transition between navigation and manipulation:
- Lead Mode: The robot identifies targets (e.g., a door handle or elevator button) using state-of-the-art vision models like NanoOWL and SAM 2. It then calculates a "Task-Aware Placement"—a position that allows the user to reach the target without the robot obstructing the workspace. Crucially, the robot uses visual servoing to align the leash, physically "pointing" the user's hand toward the target.
- Adaptation Mode: Once the system detects the user has engaged with the target, it switches to a compliant state. Using a Modified Artificial Potential Field (APF) controller, the robot dynamically adjusts its motion to stay out of the user's way while they perform the manipulation, such as pulling a door open or moving a chair.
Key Insights in Human-Robot Coordination
The effectiveness of the system stems from two primary innovations in interaction design. First, the "Pointing" Paradigm replaces complex verbal instructions (e.g., "the button is at 2 o'clock") with physical haptic feedback. By aligning the leash directly with the target, the robot provides an intuitive guide for the user's reach. Second, Adaptive Compliance ensures that the robot yields initiative to the human during the most complex part of the task. This mixed-initiative control recognizes that the human is best suited to dictate the trajectory of a swinging door or a moving chair, while the robot remains ready to resume leading once the interaction is complete.
Evaluation and Results
The system was rigorously tested with nine BVI participants across four scenarios: elevator use, pulling doors, pushing doors, and seating. These were followed by route-level field trials in realistic outdoor and indoor environments.
Key findings include:
- Enhanced Efficiency: The collaborative system was significantly faster than a traditional white cane, particularly in elevator tasks where it saved an average of 11 seconds per interaction.
- Superior Precision: In the "Locate" stage of finding handles or buttons, the full collaborative system was significantly faster than a non-adaptive robotic baseline (p < .02).
- Safety and Reliability: During route-level trials, the robotic system achieved zero collisions, whereas users with white canes averaged 0.67 collisions per run.
- User Acceptance: Participants highly rated the system's ability to announce which side to stand on and the requirement for user confirmation before resuming lead mode, indicating a high level of trust and perceived control.
Conclusion and Future Work
This work demonstrates that the future of assistive navigation lies in collaboration rather than full automation. By combining robotic precision with human dexterity, we can enable BVI individuals to navigate complex, interactive environments with greater independence. Future research will focus on expanding the system's capabilities to handle more complex social interactions, such as navigating queues, and improving vision robustness in challenging lighting conditions.
Summary updated