Abstract
This research investigates the robustness of state-of-the-art assistive robotic models by introducing environmental "disturbances"—such as dimmed lighting, glass doors, and mirrors—into virtual training simulations. By systematically challenging sensory inputs, the study reveals a significant performance gap between idealized simulations and realistic home environments. Findings show that even advanced multi-modal models (combining vision, depth, and language) suffer substantial performance degradation, particularly when faced with visual illusions like glass walls, highlighting the need for more resilient architectures in elderly care applications.
Introduction
Assistive robots designed for domestic environments, particularly for the elderly, require exceptional reliability. However, most current robotic models are developed and tested in "perfect" virtual environments that fail to account for the complexities of a real human home. While multi-modal models—integrating vision, depth, and language—generally outperform single-modality systems, their failure modes under sensory stress are not well understood. For instance, a robot relying heavily on RGB vision might mistake a reflection for a real object or fail to detect a transparent barrier. This paper systematically tests the robustness of these models by introducing controlled environmental "sabotage."
Methodology and Experimental Setup
The researchers utilized the AI2Thor virtual environment, a Unity-based simulator, to evaluate three state-of-the-art models: HLSM (Hierarchical Language-conditioned Spatial Model), FILM (Following Instructions in Language with Modular methods), and EmBERT (Embodied BERT).
The study introduced three primary disturbances to challenge the models:
- Dimmed Light: Simulating low-light conditions to test visual recognition limits.
- Glass Doors: Creating scenarios where visual paths are clear but physical movement is obstructed, testing the integration of depth sensing.
- Mirrors: Providing misleading visual information to evaluate the robot's ability to distinguish reflections from reality.
Experiments were conducted using the ALFRED (Action Learning From Realistic Environments and Directives) benchmark, which provides high-level household tasks paired with step-by-step instructions. The authors modified the "seen test set" by adding random starting positions and implementing the aforementioned disturbances.
Key Insights and Results
The introduction of a "Glass Wall" disturbance revealed critical vulnerabilities in vision-centric models. Because these agents often see "through" the glass, they repeatedly attempt to navigate through the barrier. Since the ALFRED benchmark limits agents to ten failed actions, many robots quickly exhausted their attempts and failed the task entirely.
The data suggests that additive modalities, such as combining depth scanners with RGB cameras, are crucial for safety. For the HLSM model, providing ground-truth depth information increased the Success Rate (SR) in glass-wall scenarios from 8% to 12.5%, indicating that depth perception serves as a primary defense against visual illusions.
Quantitative results across 72 episodes showed that the HLSM model was the most fragile, with its Success Rate nearly halving from 15.28% to 8.33% when faced with a glass obstacle. While the FILM model maintained its overall success rate, its Goal Condition (GC) success—the percentage of sub-goals completed—dropped by approximately 8%, indicating reduced efficiency and more frequent intermediate failures.
Conclusion and Future Work
This preliminary study demonstrates that current multi-modal models are highly susceptible to environmental disturbances common in real-world homes. Future research will expand to include mirror testing and evaluations in "unseen" floorplans to better simulate deployment in novel environments. Additionally, the researchers plan to collaborate with the Geriatronics community to identify the most critical real-world disturbances, ensuring that future robustness testing aligns with the practical needs of healthcare professionals and caregivers.
Summary updated