WiCGesture: Meta-Motion-Based Continuous Gesture Recognition With Wi-Fi

Ruiyang Gao, Wenwei Li, Jinyi Liu, Shuyu Dai, Mi Zhang, Leye Wang, Daqing Zhang

TL;DR

This paper introduces WiCGesture, a novel system for recognizing continuous in-air hand gestures using commodity Wi-Fi devices. Unlike previous methods that require pauses between gestures, WiCGesture can accurately identify gestures even when performed seamlessly, a significant step towards more natural human-computer interaction.

What Problem Does This Paper Address?

Recognizing human gestures using Wi-Fi signals has seen rapid advancements, offering a non-contact and natural way to interact with devices. However, existing Wi-Fi-based gesture recognition systems primarily focus on identifying individual, isolated gestures. They struggle to recognize continuous gestures, where users naturally perform a sequence of movements without explicit pauses between them. This limitation hinders applications like handwriting or drawing, where seamless transitions are crucial for conveying intent. The core challenge lies in segmenting and identifying gestures from an uninterrupted stream of hand movements, especially given the diverse and unpredictable nature of transitions between gestures.

Methods

The key innovation in WiCGesture is the introduction of meta motion, a novel feature designed to geometrically characterize fundamental hand movements. Instead of segmenting gestures at the gesture-transition level, WiCGesture breaks down continuous hand movements into basic "fractions" that represent distinct moving patterns. These meta motions are derived from analyzing geometric characteristics like displacement and direction changes in the Wi-Fi signal's Channel State Information (CSI).

The system comprises three main components:

  1. Preprocessing: This module cleans the raw CSI signals and uses multiple subcarriers to capture hand movements, mitigating the impact of low sampling rates and environmental noise. Techniques like CSI ratio are employed to remove phase offsets.
  2. Feature Extraction: This is where meta motions are extracted. The system estimates displacement and direction variations from the CSI data. Two types of meta motions are defined:
  3. Type 1 Meta Motion: Characterized by abrupt direction changes.
  4. Type 2 Meta Motion: Characterized by gradual rotation with fixed displacement and direction changes. The sequence of these meta motions forms a unique "fingerprint" for gestures.

  5. Recognition: A backtracking searching algorithm is used to identify gestures. The system generates reference sequences of meta motions for each gesture. It then searches for subsequences within the continuously extracted meta motion stream that match these reference sequences. This approach allows for the identification of gestures even when transitions are not perfectly segmented.

Datasets

The paper evaluates WiCGesture using two gesture sets:

  • Gesture Set S₁: Composed of digits (0-9).
  • Gesture Set S₂: Composed of mathematical symbols (Greek letters α, β, ε, η, μ, ρ).

Multiple gesture combinations were devised for evaluation on both sets, including triples, quadruples, and quintuples of gestures. The evaluation was conducted in three distinct indoor environments (office, living room, bedroom) with multiple users.

Key Insights and Ideas

  • Meta Motion as a Universal Gesture Descriptor: The concept of meta motion provides a powerful way to represent complex hand movements in a simplified, geometric manner, making it robust to variations in speed and individual execution.
  • Continuous Recognition Without Explicit Segmentation: By focusing on meta motion sequences rather than precise gesture boundaries, WiCGesture bypasses the difficult problem of accurate segmentation in continuous streams.
  • Robustness to Transitions: The meta motion approach inherently handles transitions by treating them as a sequence of basic movements, rather than a distinct event to be segmented. This is a major breakthrough for continuous gesture recognition.
  • Leveraging Commodity Wi-Fi: The system's ability to work with standard Wi-Fi devices makes it highly practical and deployable for a wide range of applications.

Key Results

WiCGesture demonstrates strong performance across various scenarios:

  • Overall Accuracy: Achieves an average accuracy of 89.6% for digit recognition (Set S₁) and 88.3% for mathematical symbol recognition (Set S₂).
  • Continuous Gesture Recognition:
    • In scenarios with paused transitions, WiCGesture achieved an average accuracy of 94.8%, slightly outperforming existing methods.
  • In scenarios with moving transitions, WiCGesture significantly outperformed other methods, which failed to recognize continuous gestures in this setting.
  • Single Gesture Recognition: Outperforms Wi-Finger, WiGesture, HandGest, and DPSense by a significant margin (e.g., 29.4% better than Wi-Finger).
  • Robustness: The system shows robustness to variations in users, speeds (up to 15 cm/s), and environments. Accuracy slightly decreases at very high speeds (20 cm/s) and longer LoS distances.

Limitations and Future Work

  • Cascading Misidentifying: The paper acknowledges that cascading misidentification (where an error in recognizing one gesture affects the next) can occur, though it is rare and typically affects only one or two subsequent gestures. They propose a preliminary approach of periodically reevaluating recognized segments in reverse order to mitigate this.
  • Specific Gesture Limitations: For very simple gestures like '1' and '7', which can be confused with transitions, WiCGesture relies on specific rules for identification.
  • Complex Character Sets: While effective for digits and symbols, recognizing highly complex character sets (like English typing) might require incorporating more distinctive features, potentially inspired by other hierarchical recognition methods.
  • Future Work: The authors suggest exploring more distinctive features for hierarchical gesture recognition and further optimizing the system for enhanced robustness and broader applicability.

Summary updated

Abstract

Recent advancements in Wi-Fi-based sensing technologies have enabled effective hand gesture recognition. However, most studies focus on single gesture recognition and fail to recognize naturally performed continuous gestures without pauses in transitions. The main challenges include diverse and uncertain transitions in continuous gesture recognition, making it difficult to segment and identify gestures from a stream of continuous hand movements. In this paper, we introduce a new method to recognize continuously performed gestures from a set of predefined gestures (e.g., digits) without requiring a pause in transitions. Instead of segmenting gestures at the gesture-transition level, we segment the stream into basic fractions that depict exclusive moving patterns of gestures. We propose a novel feature called meta motion, which geometrically characterizes different basic hand movements. Leveraging this feature, we use a back-tracking searching-based algorithm to identify gestures from the sequence of meta motions. Based on this approach, we develop a prototype system, WiCGesture, on commodity Wi-Fi devices. WiCGesture is the first system engaging in continuous gesture recognition using Wi-Fi signals. Evaluation results show that WiCGesture effectively recognizes continuous gestures from two gesture sets, significantly outperforming state-of-the-art methods.
BibTeX
@article{gao2024wicgesture,
  author = {Ruiyang Gao and Wenwei Li and Jinyi Liu and Shuyu Dai and Mi Zhang and Leye Wang and Daqing Zhang},
  title = {{WiCGesture: Meta-Motion-Based Continuous Gesture Recognition With Wi-Fi}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  doi = {10.1109/jiot.2023.3343875},
}
Figure