Development of Real-Time Hand Gesture Recognition for Tabletop Holographic Display Interaction Using Azure Kinect

Chanhwi Lee, Jaehan Kim, Seoungbae Cho, Jinwoong Kim, Jisang Yoo, Soonchul KwonSensors 2020 · DOI

TL;DR

This paper presents a real-time hand gesture recognition system designed to control a tabletop holographic display. By leveraging the Azure Kinect's depth sensing capabilities and a deep learning model called CrossInfoNet, the system enables intuitive interaction with holographic content without requiring any additional equipment, even in low-light environments.

What Problem Does This Paper Address?

Interacting with digital devices often relies on physical interfaces like keyboards, mice, or touchscreens. While gesture interaction offers a more natural and potentially accessible alternative, existing solutions face challenges. For instance, many systems require good lighting conditions or struggle with accuracy in cluttered environments. Specifically, controlling holographic displays, which often necessitate dark environments for optimal viewing, presents a unique challenge. Traditional computer vision methods that rely on color information are not suitable here. This paper addresses the need for a robust, real-time gesture recognition system that can operate effectively in low-light conditions and control holographic displays, focusing on intuitive hand gestures.

Methods

The core of the system relies on the Azure Kinect depth camera, which provides depth information essential for gesture recognition in the absence of color. The process begins with background subtraction to isolate the user and their hand from the static environment. A Region of Interest (ROI) is then defined to focus processing on the hand area.

For hand detection and joint estimation, the paper employs CrossInfoNet, a deep learning model. Unlike other models that process the entire hand image, CrossInfoNet iteratively refines palm and finger joint information through a novel multi-task information sharing mechanism. This allows for more accurate estimation of 14 hand joints.

Gestures are defined based on the relative positions and relationships of these detected hand joints. The system recognizes eight distinct gestures: basic rotations (up, down, left, right), zoom in/out, and continuous rotation (left/right). These gestures are defined using metrics like the inner product of vectors between fingers and the palm, and the distance between specific finger pairs (e.g., thumb and index finger for zoom).

The system is designed for real-time operation with a one-second delay between recognized gestures. Communication between the Azure Kinect, the processing server, and the holographic display is handled via UDP/IP sockets.

Datasets

The paper mentions training the CrossInfoNet model on the NYU hand dataset. For evaluation, experiments were conducted with 10 human subjects performing predefined gestures in the experimental setup. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Depth-centric interaction for dark environments: The paper demonstrates that robust gesture interaction is achievable using only depth information, making it ideal for environments like holographic displays that require darkness.
  • CrossInfoNet's iterative refinement: The CrossInfoNet architecture's ability to iteratively refine hand joint information by sharing features between palm and finger branches is a key innovation for accurate hand pose estimation.
  • Intuitive gesture mapping: The defined gestures (e.g., swiping for rotation, pinching/spreading for zoom) are designed to be intuitive and easily learned by users, mimicking real-world interactions.
  • System integration for holographic control: The successful integration of gesture recognition with a tabletop holographic display showcases a practical application for this technology.

Key Results

The system achieved high performance across various distances:

  • At 35-50 cm: Precision of 0.99747, Recall of 0.98872, and an F1 score of 0.99307.
  • At 50-60 cm: Precision of 0.99747, Recall of 0.95500, and an F1 score of 0.97577.
  • At 60-70 cm: Precision of 0.99747, Recall of 0.90875, and an F1 score of 0.95104.

Specific gestures like "Up" and "Zoom In/Out" showed very high F1 scores (e.g., 98.98% and 99.49% respectively at 35-50 cm). The system successfully controlled holographic image rotation, zooming, and continuous rotation based on hand gestures.

Limitations and Future Work

  • User-specific hand size: The system struggles with users who have smaller hands, leading to undetected gestures and false positives. The authors suggest dynamically adjusting thresholds to accommodate different hand sizes.
  • Distance sensitivity: Performance degrades as the distance between the user and the camera increases, with significant drops in accuracy beyond 70 cm.
  • Wrist bending: The system had difficulty accurately recognizing gestures involving significant wrist bending, particularly for rightward rotation.
  • Limited subject pool: The experiment involved only 10 subjects, which may limit the generalizability of the results. Future work should involve a larger and more diverse group of participants.
  • Angle limitations: The system's performance is affected if the user and camera are not aligned or if the user is significantly off-axis (more than 30 degrees).
  • Future directions: The authors propose improving the system to better handle small hands, adjusting gesture thresholds for specific movements like wrist bending, and expanding the repertoire of recognized gestures to offer more comprehensive control over holographic displays. They also envision applying this depth-only gesture interaction to smart home and automotive applications.

Summary updated

BibTeX
@article{lee2020development,
  author = {Chanhwi Lee and Jaehan Kim and Seoungbae Cho and Jinwoong Kim and Jisang Yoo and Soonchul Kwon},
  title = {{Development of Real-Time Hand Gesture Recognition for Tabletop Holographic Display Interaction Using Azure Kinect}},
  journal = {Sensors},
  year = {2020},
  doi = {10.3390/s20164566},
}
Figure