Review of Vision-Based Environmental Perception for Lower-Limb Exoskeleton Robots

Chen Wang, Zhongcai Pei, Yanan Fan, Shuang Qiu, Zhiyong TangBiomimetics 2024 · DOI

TL;DR

This paper reviews the state-of-the-art in vision-based environmental perception for lower-limb exoskeletons, highlighting its crucial role in enhancing safety, stability, and human-machine interaction. It delves into key technologies like datasets, terrain detection, and adaptive gait planning, while also identifying current limitations and future research directions.

What Problem Does This Paper Address?

Lower-limb exoskeletons, whether for medical rehabilitation or power assistance, are increasingly sophisticated. However, their ability to navigate complex and dynamic environments safely and effectively is often hindered by a lack of robust environmental perception. This paper addresses the critical need for exoskeletons to "see" and understand their surroundings, moving beyond simple pre-programmed movements or basic human intent recognition. The challenge lies in integrating visual information into the exoskeleton's control loop to enable autonomous decision-making, improve stability, and create a more seamless human-machine interaction, especially in unpredictable real-world scenarios.

Methods

The paper categorizes visual sensors into passive (e.g., RGB cameras) and active (e.g., structured light, Time-of-Flight, LiDAR) types, discussing their pros and cons for wearable applications. It also explores the integration of these sensors into a three-layered control system: high-level (perception and prediction), middle-level (kinematic modeling), and low-level (motor control). The core of the paper focuses on key technologies for environmental perception, including:

  • Environment Classification: Using deep learning models (CNNs) to categorize the surrounding environment.
  • Terrain Detection: Specifically focusing on identifying stairs and ramps, employing both traditional image processing and deep learning approaches.
  • Obstacle Detection: Adapting methods from autonomous driving to identify objects that pose a risk.
  • Environment-Oriented Adaptive Gait Planning: Developing strategies for the exoskeleton to adjust its walking pattern based on perceived environmental features, often using techniques like Dynamic Movement Primitives (DMPs) and Model Predictive Control (MPC).

Datasets

The paper discusses several datasets crucial for training and evaluating environmental perception systems for exoskeletons:

  • ExoNet dataset: A large-scale dataset (922,790 images) for environment classification, capturing diverse walking environments.
  • Stair Recognition Datasets: Including datasets built upon ExoNet (e.g., Kurbis and Laschowski's dataset) and dedicated stair detection datasets (e.g., Patil et al., Rekhawar et al.).
  • RGB-D Datasets: Such as Zhu's dataset, which combines RGB and depth information for improved performance in complex scenes.
  • Universal Datasets: PASCAL, COCO, ADE20K, and NYUV2 are mentioned as being used or re-annotated for obstacle detection tasks.

The paper notes that while datasets for environment classification are abundant, those for object detection and semantic segmentation are often smaller, potentially limiting model generalization.

Key Insights and Ideas

  • Vision is Essential for Advanced Exoskeletons: The paper strongly emphasizes that for both power-assisted and medical rehabilitation exoskeletons, visual perception is no longer a luxury but a necessity for safe and intelligent operation.
  • Dual Role of Vision: For power-assisted exoskeletons, vision helps anticipate upcoming environments for smoother motion tracking. For medical exoskeletons, it's crucial for extracting precise terrain parameters to enable safe and reliable navigation.
  • Deep Learning Dominance: The review highlights the shift from traditional image processing to deep learning methods, particularly Convolutional Neural Networks (CNNs), for most environmental perception tasks.
  • Challenges in Real-World Deployment: Despite advancements, the paper points out that current systems still struggle with robustness in diverse lighting conditions, complex textures, and the need for real-time performance on resource-constrained wearable devices.
  • Human-in-the-Loop is Key: Unlike autonomous driving, exoskeleton control must prioritize the human user. Environmental perception should aim to assist, not replace, human control, considering subjective feelings and promoting recovery.

Key Results

The paper doesn't present new experimental results but rather synthesizes findings from existing research. It highlights:

  • Deep Learning Model Performance: EfficientNetB0 is noted for achieving high classification accuracy, VGG16 for fast inference, and MobileNetV2 for a good NetScore in environment classification.
  • StairNet Series Effectiveness: The StairNet series demonstrates the potential of deep learning for end-to-end stair detection, offering improvements in accuracy and speed over traditional methods.
  • RGB-D Fusion Benefits: RGB-D datasets and fusion methods show promise in improving performance in challenging visual conditions.
  • Installation Position Matters: For lower-limb exoskeletons, chest and waist mounting positions are identified as optimal for stable vision data synchronized with user movement.

Limitations and Future Work

The authors identify several limitations and propose future directions:

  • Dataset Limitations: Existing datasets, while growing, often lack sufficient diversity in gait-transition scenarios and may not adequately represent all daily walking environments. There's a need for more comprehensive scene-understanding datasets for outdoor environments.
  • Human-Exoskeleton Interaction: Effectively combining signals from various sensors (EMG, vision, etc.) with different triggering times and modalities remains a challenge. Adjusting the role of humans in the loop and developing datasets that capture this interaction is crucial.
  • Real-Time Performance: Achieving real-time performance for complex perception tasks on power-constrained wearable devices is an ongoing challenge, especially for methods relying on large point cloud data.
  • Gait Planning: Developing truly biomimetic and online-adjustable gait planning methods is the ultimate goal. This requires better reference gait data (especially for rehabilitation) and robust methods for handling environmental changes and human disturbances.
  • Obstacle Detection for Rehabilitation: Adapting obstacle detection from autonomous driving to the specific needs of rehabilitation exoskeletons, where user assistance and risk assessment are paramount, requires further research.

Summary updated

Abstract

The exoskeleton robot is a wearable electromechanical device inspired by animal exoskeletons. It combines technologies such as sensing, control, information, and mobile computing, enhancing human physical abilities and assisting in rehabilitation training. In recent years, with the development of visual sensors and deep learning, the environmental perception of exoskeletons has drawn widespread attention in the industry. Environmental perception can provide exoskeletons with a certain level of autonomous perception and decision-making ability, enhance their stability and safety in complex environments, and improve the human-machine-environment interaction loop. This paper provides a review of environmental perception and its related technologies of lower-limb exoskeleton robots. First, we briefly introduce the visual sensors and control system. Second, we analyze and summarize the key technologies of environmental perception, including related datasets, detection of critical terrains, and environment-oriented adaptive gait planning. Finally, we analyze the current factors limiting the development of exoskeleton environmental perception and propose future directions.
BibTeX
@inproceedings{wang2024review,
  author = {Chen Wang and Zhongcai Pei and Yanan Fan and Shuang Qiu and Zhiyong Tang},
  title = {{Review of Vision-Based Environmental Perception for Lower-Limb Exoskeleton Robots}},
  booktitle = {Biomimetics},
  year = {2024},
  doi = {10.3390/biomimetics9040254},
}
Figure