ExoNet Database: Wearable Camera Images of Human Locomotion Environments

Brock Laschowski, William J. McNally, Alexander Wong, John McPhee

Introduction

Millions of individuals worldwide rely on lower-limb exoskeletons and prostheses to regain or enhance their mobility. A critical challenge in the development of these assistive devices is "locomotion mode recognition"—the system's ability to identify whether a user is walking on level ground, ascending stairs, or preparing to sit.

Traditionally, these devices have relied on inertial measurement units (IMUs) or electromyography (EMG) sensors. While effective, these methods are inherently reactive, detecting changes only after a movement has begun. This lag can result in instability or falls. To move toward proactive control, researchers are turning to computer vision, allowing devices to anticipate environmental changes before the user encounters them. The ExoNet database was created to fill the gap in large-scale, open-source datasets required to train the deep learning models necessary for this vision-based adaptation.

Methods and Data Collection

The researchers developed a robust pipeline for data collection and hierarchical labeling to ensure the dataset's utility for real-world robotic control.

Overview of ExoNet data collection hardware (A), sample environment images (B), and the hierarchical labeling architecture (C).

Hardware and Environment

Data was collected using a single subject wearing a chest-mounted smartphone (iPhone XS Max). The chest mount was selected over limb-mounting to provide a stable, consistent field-of-view and a high-resolution (1280 × 720) RGB perspective of the path ahead. Over 52 hours of footage were recorded across diverse indoor and outdoor settings, capturing seasonal variations such as snow, fallen leaves, and different lighting conditions to ensure model robustness.

Hierarchical Labeling Architecture

To translate visual data into actionable robotic commands, the team implemented a 12-class hierarchical labeling system. This architecture categorizes images based on three levels:

  1. Environment: The broad setting (e.g., Incline Stairs, Level Ground, Door/Wall).
  2. State: Distinguishing between "Steady" movement and "Transition" phases.
  3. Condition: Specific details, such as level ground leading to a seat or stairs transitioning back to level ground.

While the video was captured at 30 Hz, the images were downsampled to 5 frames per second for manual annotation to maximize visual diversity and minimize redundancy.

Dataset Characteristics

ExoNet represents a significant leap in scale compared to previous efforts in the field. The database contains approximately 5.6 million total images, with 922,790 individually labeled images. For comparison, previous datasets often contained fewer than 10,000 images and limited class diversity. By providing nearly a million labeled frames at 720p resolution, ExoNet offers a rich feature set for training complex deep convolutional neural networks (CNNs).

Key Insights

  • Functional Labeling: Unlike general-purpose computer vision datasets that might distinguish between grass and pavement, ExoNet groups them as "Level Ground." This reflects the functional requirements of robotics, where the control logic for walking on both surfaces is identical.
  • The Transition Window: A major innovation is the explicit focus on "Transition" states. Identifying the moment a user moves toward a staircase is the critical window for a high-level controller to trigger a mode switch safely.
  • RGB over Depth: The authors chose high-resolution RGB over depth cameras due to the latter's limitations in direct sunlight, restricted range (1–2m), and high power consumption, which are impractical for untethered outdoor use.

Limitations and Future Directions

While ExoNet is the largest dataset of its kind, it currently features data from a single subject. Future iterations will aim to include data from users with varying heights, gaits, and walking speeds. Additionally, while the dataset is currently 2D, future work may involve fusing these RGB images with onboard IMU data to better synchronize environmental "intent" with actual physical movement. This data report serves as a foundation for subsequent research into benchmark classification accuracies using state-of-the-art deep learning models.

Summary updated

Abstract

1 IntroductionHundreds of millions of individuals worldwide have mobility impairments resulting from degenerative aging and neuro-musculoskeletal disorders like spinal cord injury, osteoarthritis, Parkinson’s disease, and cerebral palsy (Grimmer et al., 2019). Fortunately, newly-developed powered lower-limb exoskeletons and prostheses can allow otherwise wheelchair-bound seniors and rehabilitation patients to perform movements that involve net positive mechanical work (e.g., climbing stairs and standing from a seated position) using onboard actuators and intelligent control systems (Krausz and Hargrove, 2019; Laschowski and Andrysek, 2018; Tucker et al., 2015; Young and Ferris, 2017; Zhang et al., 2019a). Generally speaking, the high-level controller recognizes the patient’s locomotion mode (intention) by analyzing real-time measurements from wearable sensors using machine learning algorithms. The mid-level controller then translates the locomotion intentions into mode-specific reference trajectories. This control level typically comprises a finite state machine, which implements a discrete parametrized control law (e.g., joint position or mechanical impedance control) for each different locomotion mode. Finally, the low-level controller tracks the reference trajectories and minimizes the signal error by modulating the device actuators using feedforward and feedback control loops (Krausz and Hargrove, 2019; Laschowski and Andrysek, 2018; Tucker et al., 2015; Young and Ferr...
BibTeX
@article{laschowski2020exonet,
  author = {Brock Laschowski and William J. McNally and Alexander Wong and John McPhee},
  title = {{ExoNet Database: Wearable Camera Images of Human Locomotion Environments}},
  journal = {Frontiers Robotics AI},
  year = {2020},
  doi = {10.3389/frobt.2020.562061},
}
Figure