Wheelchair Behavior Recognition for Visualizing Sidewalk Accessibility by Deep Neural Networks

Takumi Watanabe, Hiroki Takahashi, Goh Sato, Yusuke Iwasawa, Yutaka Matsuo, Ikuko Eguchi YairiarXiv 2021 · OpenReview

Abstract

This paper presents an automated system for mapping sidewalk accessibility by analyzing wheelchair vibrations captured via smartphone accelerometers. By combining supervised deep learning with weakly supervised and self-supervised techniques, the researchers demonstrate how to identify road features like curbs and slopes without relying on expensive, manual human annotations. The system achieves high accuracy in road condition classification and effectively identifies physical barriers through anomaly detection, offering a scalable solution for urban navigation.

Introduction

Navigating urban environments is a significant challenge for people with mobility difficulties. While accessibility maps exist, they usually rely on manual surveys by experts or crowdsourced data from volunteers—both of which are difficult to scale across entire cities. Furthermore, "accessibility" requirements vary significantly between individuals; a Paralympic athlete and an elderly person have very different needs. To build truly useful navigation tools, objective data on the physical state of the road (e.g., the exact height of a curb or the angle of a slope) is required. This research aims to automate this data collection by turning every wheelchair into a mobile sensor.

Methodology

The researchers proposed a three-tiered approach using Deep Neural Networks to process vibration data:

  1. Supervised Learning (CNN): A Convolutional Neural Network was trained to classify four specific road conditions: Slopes, Curbs, Tactile Indicators, and standard flat pavement.
  2. Weakly Supervised Learning (PosNet): To reduce the need for manual labels, GPS data was used as "weak" labels. The network was tasked with predicting which 5m grid cell the wheelchair was in, forcing it to learn unique "vibration signatures" of specific locations.
  3. Self-Supervised Learning (ConvVAE): A Convolutional Variational Autoencoder was employed to detect anomalies. By training the model to reconstruct "normal" driving data, reconstruction error was used as a proxy for road difficulty; high error rates typically pointed to curbs or unusual obstacles.
Overview of the automated system for sensing, analyzing, and visualizing sidewalk accessibility.

Experimental Setup

The study utilized a custom-built dataset involving 9 wheelchair users (6 manual, 3 electric) navigating a 1.4 km specified route around Yotsuya Station in Tokyo. Hardware consisted of an iPod Touch sampling triaxial acceleration at 50 Hz. The route included various pavement types, including gravel, tile, block, and concrete, providing a diverse set of vibration profiles for model training.

Map of the experimental route in Tokyo and visual examples of the target road condition classes.

Results and Discussion

The CNN model achieved a 71.3% Macro F-score and 86.4% accuracy, significantly outperforming traditional machine learning methods using hand-crafted features.

A key insight was the effectiveness of "Location as a Label." The weakly supervised PosNet approach outperformed the fully supervised CNN when the amount of labeled data dropped below 10%, proving the system's viability for real-world deployment where labels are scarce. Furthermore, visualization of the VAE reconstruction error clearly highlighted specific curb points and construction sites on the map, validating its use as an automated barrier detector.

Conclusion

This research demonstrates that deep learning can effectively automate the mapping of sidewalk accessibility using low-cost smartphone sensors. By leveraging weakly supervised and self-supervised techniques, the system reduces the burden of manual data annotation. Future work will focus on expanding the diversity of the participant pool and exploring hybrid models like CNN-LSTMs to further improve time-series recognition performance.

Summary updated

Abstract

This paper introduces our methodology to estimate sidewalk accessibilities from wheelchair behavior via a triaxial accelerometer in a smartphone installed under a wheelchair seat. Our method recognizes sidewalk accessibilities from environmental factors, e.g. gradient, curbs, and gaps, which influence wheelchair bodies and become a burden for people with mobility difficulties. This paper developed and evaluated a prototype system that visualizes sidewalk accessibility information by extracting knowledge from wheelchair acceleration using deep neural networks. Firstly, we created a supervised convolutional neural network model to classify road surface conditions using wheelchair acceleration data. Secondly, we applied a weakly supervised method to extract representations of road surface conditions without manual annotations. Finally, we developed a self-supervised variational autoencoder to assess sidewalk barriers for wheelchair users. The results show that the proposed method estimates sidewalk accessibilities from wheelchair accelerations and extracts knowledge of accessibilities by weakly supervised and self-supervised approaches.
BibTeX
@misc{watanabe2021wheelchair,
  author = {Takumi Watanabe and Hiroki Takahashi and Goh Sato and Yusuke Iwasawa and Yutaka Matsuo and Ikuko Eguchi Yairi},
  title = {{Wheelchair Behavior Recognition for Visualizing Sidewalk Accessibility by Deep Neural Networks}},
  year = {2021},
  eprint = {2101.03724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.03724},
}
Figure