A multimodal gesture recognition dataset for desktop human-computer interaction

Qi Wang, Fengchao Zhu, Guangming Zhu, Liang Zhang, Ning Li, Eryang Gao

Introduction: Solving the "Gorilla Arm" Problem

Most existing gesture recognition datasets rely on front-facing cameras. While effective for some scenarios, this setup presents two major drawbacks for desktop users: ergonomic fatigue and visual interference. Users must often raise their hands to a higher position to stay in the camera's field of view, leading to "gorilla arm" syndrome during prolonged use. Furthermore, front-facing cameras often capture the user's face, raising privacy concerns and introducing background occlusion issues.

To address these challenges, researchers have introduced GR4DHCI, a large-scale multimodal dataset specifically designed for top-down (down-view) gesture recognition. This setup allows users to perform gestures naturally and comfortably over a desk-mounted sensor like the Leap Motion Controller (LMC).

Desktop setup for down-view gesture recognition using a Leap Motion Controller.

The GR4DHCI Dataset

The GR4DHCI dataset is tailored for the top-down perspective and provides a robust foundation for desktop HCI research:

  • Scale: 7,339 video samples and 382,447 frames.
  • Modality: Synchronized Stereo Infrared (IR) imagery (640x480) and 25-point 3D skeletal sequences.
  • Diversity: Data was collected from 16 subjects across 27 different hand positions (varying height, depth, and horizontal placement) to ensure the model generalizes beyond a single "sweet spot" above the sensor.
  • Gesture Vocabulary: The dataset includes seven common UI navigation categories: left swipe, right swipe, up swipe, down swipe, vertical move, horizontal move, and zoom.
The seven gesture categories in the GR4DHCI dataset visualized via infrared imagery.

Methodology: Dual-Modality and Fine-Grained Classification

The researchers proposed a dual-modality approach that leverages the strengths of both Stereo IR imagery and 3D skeletal data. Their methodology utilizes Topology-Learnable Graph Convolutional Networks (TL-GCN) for skeletal data and a combination of Res3D and ConvLSTM for the infrared stream.

The paper's primary algorithmic contribution is the Fine-Grained Classification Block (FGCB). This refinement layer acts as a "sanity check" by combining:

  1. Hand Shape Classification: A lightweight MobileNet identifies the general hand shape category.
  2. Trajectory Matching: Using Longest Common Subsequence (LCSS) and Edit Distance on Real sequence (EDR), the system compares the performed gesture's path against predefined ideal trajectories.

If the FGCB’s high-confidence recommendation differs from the baseline model’s prediction, the system corrects the output to the recommended gesture, significantly reducing temporal confusion between similar moves.

Results and Key Insights

The inclusion of the FGCB provided a substantial performance boost across both modalities:

  • Skeletal Modality (TL-GCN): Accuracy improved from 93.50% to 96.14%.
  • Infrared Modality (Res3D+ConvLSTM): Accuracy increased from 82.57% to 90.32%.

Key insights from the study highlight that while skeletal data is generally more robust against noise, the IR stream provides essential hand-shape details that skeletal points alone might miss. By focusing on a "down-view," the authors prioritize user comfort, allowing gestures to be performed in the natural resting space above a keyboard or mouse.

Limitations and Future Directions

While GR4DHCI is a significant step forward, the authors note several areas for future work. The current gesture vocabulary is limited to seven categories; a broader set would be required for more complex applications. Additionally, expanding the subject pool to include more diverse hand sizes and ages would improve generalization. Finally, while the FGCB components are lightweight, further research into end-to-end latency is critical for ensuring the real-time responsiveness required for seamless human-computer interaction.

Summary updated

Abstract

Gesture recognition is an indispensable component of natural and efficient human-computer interaction technology, particularly in desktop-level applications, where it can significantly enhance people's productivity. However, the current gesture recognition community lacks a suitable desk-top-level (top-view perspective) dataset for lightweight gesture capture devices. In this study, we have established a dataset named GR4DHCI. What distinguishes this dataset is its inherent naturalness, intuitive characteristics, and diversity. Its primary purpose is to serve as a valuable resource for the development of desktop-level portable applications. GR4DHCI comprises over 7,000 gesture samples and a total of 382,447 frames for both Stereo IR and skeletal modalities. We also address the variances in hand positioning during desktop interactions by incorporating 27 different hand positions into the dataset. Building upon the GR4DHCI dataset, we conducted a series of experimental studies, the results of which demonstrate that the fine-grained classification blocks proposed in this paper can enhance the model's recognition accuracy. Our dataset and experimental findings presented in this paper are anticipated to propel advancements in desktop-level gesture recognition research.
BibTeX
@misc{wang2024multimodal,
  author = {Qi Wang and Fengchao Zhu and Guangming Zhu and Liang Zhang and Ning Li and Eryang Gao},
  title = {{A multimodal gesture recognition dataset for desktop human-computer interaction}},
  year = {2024},
  eprint = {2401.03828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.03828},
}
Figure