Deep Fisher Discriminant Learning for Mobile Hand Gesture Recognition.

Chunyu Xie, Ce Li, Baochang Zhang, Chen Chen, Jungong Han

TL;DR

This paper introduces F-BLSTM and F-BGRU, two deep learning architectures that integrate the classical Fisher criterion into recurrent neural networks to enhance mobile hand gesture recognition. By explicitly minimizing intra-class variance and maximizing inter-class variance during training, these models achieve state-of-the-art accuracy on a newly released large-scale mobile gesture dataset.

What Problem Does This Paper Address?

Hand gesture recognition using mobile sensors (accelerometers and gyroscopes) is a cornerstone of modern human-computer interaction. However, it remains a "noisy" problem. Models must contend with signal intensity differences (intense vs. weak movements), temporal variations (slow vs. fast execution), and physical differences between users.

While standard deep learning models like Long Short-Term Memory (LSTM) networks are excellent at handling sequential data, they often rely solely on the softmax loss function. Softmax is great for classification but doesn't necessarily force the network to learn a feature space where different classes are widely separated and the same classes are tightly clustered. This lack of discriminative power can lead to errors when gestures look similar or are performed inconsistently.

Methods

The authors propose a hybrid approach that combines the temporal modeling of Bidirectional LSTMs (BLSTM) and Bidirectional Gated Recurrent Units (BGRU) with the discriminative power of the Fisher criterion.

  1. Architecture: The system takes raw 3-axis accelerometer and 3-axis gyroscope data, preprocesses it (filtering and normalization), and feeds it into a bidirectional recurrent layer. A mean pooling layer then aggregates these temporal features.
  2. The Fisher Loss: The core innovation is the modified loss function: $$\mathcal{L} = \mathcal{L}_s + \theta \mathcal{L}_f$$ Where $\mathcal{L}_s$ is the standard softmax loss and $\mathcal{L}_f$ is the Fisher criterion. The Fisher term specifically penalizes the distance between a sample and its class mean while rewarding the distance between different class means.

  3. Optimization: The authors introduce a strategy to update class means ($\mu$) iteratively during backpropagation, ensuring the feature space remains organized as the network learns.

The architecture of F-BLSTM, showing how the Fisher criterion is integrated into the loss function.

Datasets

The study evaluates the proposed models across three benchmarks:

  • Mobile Gesture Database (MGD): A new, large-scale dataset collected by the authors using a Huawei device. It contains 5,547 sequences of 12 gestures (Arabic numerals and English letters) performed by 32 participants.
  • BUAA Mobile Gesture Database: A smaller benchmark with 1,120 samples of 8 gestures.
  • SmartWatch Gesture Database: A dataset featuring 20 different gestures performed by 8 users using a first-generation Sony SmartWatch.

Key Insights and Ideas

  • Classical Stats Meets Deep Learning: The "aha moment" here is that deep learning doesn't have to reinvent the wheel. Classical statistical methods like Fisher Linear Discriminant Analysis can be "plugged in" as loss constraints to guide the network toward more robust representations.
  • Better Feature Clustering: Visualization using t-SNE reveals that while standard BLSTMs produce somewhat overlapping clusters, F-BLSTM produces highly distinct, tight clusters. This makes the final classification task much easier for the model.
  • Efficiency in Training: Interestingly, adding the Fisher criterion doesn't just improve accuracy; it also speeds up convergence. The models reach lower error rates in fewer iterations compared to their standard counterparts.

Key Results

The F-variants consistently outperformed the baseline models across all datasets:

  • On the MGD Dataset: F-BGRU achieved an overall accuracy of 99.15%, compared to 98.38% for the standard BGRU and 97.05% for the standard BLSTM.
  • Robustness to Small Data: On the smaller BUAA dataset, the Fisher criterion helped the models converge faster and achieve higher accuracy (99.25% for F-BGRU), proving its effectiveness even when training data is limited.
  • SmartWatch Performance: Even with lower-frequency sensor data (Sony SmartWatch), the F-BGRU maintained a high accuracy of 97.40%.

Limitations and Future Work

While the results are impressive, the paper notes that the performance is sensitive to three hyperparameters: $\theta$ (the weight of the Fisher loss), $\alpha$ (the update rate of class means), and $\delta$ (the discriminative factor). Finding the "sweet spot" for these parameters requires careful tuning.

The authors suggest that future work could explore more complex gesture sets or investigate how these discriminative deep models perform in real-time, low-power mobile environments where computational resources are at a premium.

Summary updated

Abstract

Gesture recognition is a challenging problem in the field of biometrics. In this paper, we integrate Fisher criterion into Bidirectional Long-Short Term Memory (BLSTM) network and Bidirectional Gated Recurrent Unit (BGRU),thus leading to two new deep models termed as F-BLSTM and F-BGRU. BothFisher discriminative deep models can effectively classify the gesture based on analyzing the acceleration and angular velocity data of the human gestures. Moreover, we collect a large Mobile Gesture Database (MGD) based on the accelerations and angular velocities containing 5547 sequences of 12 gestures. Extensive experiments are conducted to validate the superior performance of the proposed networks as compared to the state-of-the-art BLSTM and BGRU on MGD database and two benchmark databases (i.e. BUAA mobile gesture and SmartWatch gesture).
BibTeX
@article{xie2017deep,
  author = {Chunyu Xie and Ce Li and Baochang Zhang and Chen Chen and Jungong Han},
  title = {{Deep Fisher Discriminant Learning for Mobile Hand Gesture Recognition.}},
  journal = {Pattern Recognition},
  year = {2017},
  doi = {10.1016/j.patcog.2017.12.023},
}
Figure