Text to Blind Motion

Hee Jae Kim, Kathakoli Sengupta, Masaki Kuribayashi, Hernisa Kacorri, Eshed Ohn-Bar

Most of us take for granted that an autonomous vehicle or a social robot will "understand" how we move. But what if your movement patterns don't look like the "standard" data these systems were trained on? In a new paper appearing at NeurIPS 2024, researchers from Boston University, Waseda University, and the University of Maryland highlight a glaring bias in human motion modeling: the near-total absence of data from individuals with disabilities.

To address this, the authors introduce BlindWays, the first multimodal 3D motion benchmark specifically featuring blind pedestrians navigating real-world urban environments. By providing high-fidelity motion data paired with rich textual descriptions, the paper demonstrates that current state-of-the-art motion models fail to generalize to the unique navigation strategies of the blind, posing significant safety risks for future autonomous systems.

The Problem: "Sighted Bias" in Motion Modeling

Current 3D human motion datasets, such as AMASS or HumanML3D, are overwhelmingly biased toward sighted individuals. However, blind pedestrians exhibit distinct mobility characteristics. They may "veer" from straight paths in open spaces, use rhythmic cane-sweeping motions, or pause at curbs to explore tactile cues.

Because these behaviors are "out-of-distribution" for standard models, technologies like autonomous vehicles (AVs) might misinterpret a blind person’s intent or fail to predict their path accurately. This isn't just a technical gap; it’s a safety and equity issue. If an AV cannot recognize the subtle coordination between a person and their white cane, it cannot operate safely in a truly diverse society.

Methods and Data Collection

To capture these nuances, the researchers developed a rigorous data collection and annotation pipeline designed specifically for the blind navigation context.

Overview of the BlindWays data collection and annotation pipeline, including route mapping, egocentric video, motion capture, and multi-level language descriptions.
  • Hardware and Tracking: Participants wore an Xsens motion capture suit (utilizing 18 Inertial Measurement Units or IMUs) to track body joints. Crucially, the researchers also placed a sensor on the mobility aid (cane or guide dog) to track it as an extension of the user.
  • Multi-View Context: The team recorded third-person video via smartphone and egocentric video via a chest-mounted GoPro to provide annotators with complete environmental context.
  • Two-Level Annotation: Unlike standard datasets that use simple action labels (e.g., "walking"), BlindWays provides rich, multi-tiered text:
  • High-level descriptions: Summarizing intent (e.g., "approaching a crosswalk").
  • Low-level descriptions: Detailing mechanics (e.g., "takes six small steps... uses a combination of sweeping and tapping").

The resulting BlindWays dataset includes 1,029 motion clips, approximately 0.6 million human poses, and 2,058 paired text descriptions, collected from 11 blind or low-vision individuals navigating 8 complex urban routes.

Key Insights and Results

The authors benchmarked state-of-the-art generative models (like MotionGPT and HumanML3D) to see how well they handled this new domain, revealing several critical insights:

  • The Realism Gap: Models trained only on standard sighted data (like Motion-X) produced "unrealistic" and "jittery" motions when asked to generate blind navigation, yielding poor Frechet Inception Distance (FID) scores between 11.2 and 15.0. When trained on BlindWays, FID dropped significantly to 3.3–5.1, showing a much closer match to real-world blind motion.
  • The Importance of the "Aid" Joint: The white cane is not just an object; it is an extension of the person’s sensory system. Training on BlindWays led to an 8% improvement in predicting arm movements and a massive 16.5% improvement in predicting the motion of the mobility aid compared to models trained on general datasets.
  • Pre-training Limitations: While pre-training on large-scale sighted datasets helps with general human movement, it introduces a "bias shift." This makes it harder for models to capture the specific, subtle movements of blind pedestrians unless they are heavily fine-tuned on domain-specific data.
  • Text-to-Motion Retrieval: The accuracy of matching text to the correct motion (R-Top1) was notably higher for models trained on the new dataset, proving that nuanced language descriptions are essential for the model to truly "understand" the motion.

Limitations and Future Work

The authors are transparent about the study's current scale. While 11 participants is a strong start, it is not a replacement for massive-scale data. Furthermore, the high cost of IMU suits (often exceeding $6,500) limits the ability of smaller labs to contribute to similar datasets.

Future work aims to expand the diversity of the participant pool—incorporating different ages, heights, and backgrounds—and to explore lower-cost, vision-based capture methods that could scale more easily. Ultimately, the goal of BlindWays is to integrate these inclusive models into real-world AV stacks, ensuring that the "smart" cities of the future are safe and accessible for everyone.

Summary updated

Abstract

People who are blind perceive the world differently than those who are sighted, which can result in distinct motion characteristics. For instance, when crossing at an intersection, blind individuals may have different patterns of movement, such as veering more from a straight path or using touch-based exploration around curbs and obstacles. These behaviors may appear less predictable to motion models embedded in technologies such as autonomous vehicles. Yet, the ability of 3D motion models to capture such behavior has not been previously studied, as existing datasets for 3D human motion currently lack diversity and are biased toward people who are sighted. In this work, we introduce BlindWays, the first multimodal motion benchmark for pedestrians who are blind. We collect 3D motion data using wearable sensors with 11 blind participants navigating eight different routes in a real-world urban setting. Additionally, we provide rich textual descriptions that capture the distinctive movement characteristics of blind pedestrians and their interactions with both the navigation aid (e.g., a white cane or a guide dog) and the environment. We benchmark state-of-the-art 3D human prediction models, finding poor performance with off-the-shelf and pre-training-based methods for our novel task. To contribute toward safer and more reliable systems that can seamlessly reason over diverse human movements in their environments, our text-and-motion benchmark is available at https://blindways.github.io.
BibTeX
@inproceedings{kim2024text,
  author = {Hee Jae Kim and Kathakoli Sengupta and Masaki Kuribayashi and Hernisa Kacorri and Eshed Ohn-Bar},
  title = {{Text to Blind Motion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure