Acoustic-based recognition of head gestures accompanying speech

Akira Sasou, Yasuharu Hashimoto, Katsuhiko SakaueInterspeech 2010

TL;DR

This paper introduces a novel method for recognizing head gestures solely through acoustic signals, using a microphone array to track mouth position. By applying Higher-Order Local Cross Correlation (HLCC) to the mouth trajectory, they achieve impressive head gesture recognition accuracy, outperforming traditional HMM-based approaches without relying on visual data.

What Problem Does This Paper Address?

Communicating effectively involves more than just spoken words; non-verbal cues like head movements play a crucial role. These gestures can convey symbolic meanings (like "yes" or "no") and also subtly emphasize aspects of speech, such as stress and prosody. While previous research often relied on visual information (like video cameras) to track head movements, this paper tackles the challenge of recognizing these gestures using only acoustic data. This opens up possibilities for systems that can understand head gestures in environments where visual input is limited or unavailable, or for applications where integrating visual sensors is impractical.

Methods

The core of the proposed method lies in two key components:

  1. Acoustic Localization of Mouth Position: The system uses a specialized three-axis microphone array, originally designed for noise-robust speech recognition. This array is capable of localizing sound sources in 3D space. The authors leverage the fact that the localized position of a user's voice is a good proxy for their mouth position. They employ the MUltiple SIgnal Classification (MUSIC) method for this localization. A "User Utterance Area" (UUA) is defined in front of the array; sounds originating within this area are considered user utterances, while those outside are treated as noise. This acoustic localization is performed without any visual input.
  2. Higher-Order Local Cross Correlation (HLCC) for Trajectory Recognition: Once the mouth position is tracked over time, forming an "utterance trajectory," the system extracts features using HLCC. This involves:
  3. Bias Elimination and Normalization: The raw trajectory data is first de-biased and then normalized in time.
  4. HLCC Calculation: The HLCC is computed on segments of this normalized trajectory. This technique captures local patterns and their relationships within the trajectory, extending beyond simple cross-correlation.
  5. Feature Vector Construction: The computed HLCCs from various local patterns are concatenated to form a feature vector that represents the head gesture.

For recognition, a subspace method is employed. Each head gesture is represented by a subspace learned from training data. During recognition, the HLCC feature vector is projected onto these learned subspaces, and the gesture corresponding to the subspace with the maximum projection is selected.

Datasets

The paper does not explicitly mention a publicly released dataset. Instead, the authors collected their own data for evaluation:

  • Head Gesture Data: Nineteen different types of head gestures were collected from five subjects. These included gestures for drawing numerals 0-9, eight directional movements, and random head movements. Each gesture was performed five times by each subject.
  • Noise Robustness Data: Clean utterances from three subjects were recorded separately from surrounding noises. These were then mixed at various Signal-to-Noise Ratios (SNRs) ranging from 20dB to -10dB to evaluate the microphone array's performance in noisy environments.

Key Insights and Ideas

  • Acoustic-Only Head Gesture Recognition is Feasible: The most significant insight is that head gestures can be reliably recognized using only sound, bypassing the need for cameras. This is a crucial step towards more versatile and unobtrusive human-computer interaction systems.
  • Mouth Position as a Proxy for Head Movement: The paper demonstrates that tracking the acoustic source of speech (i.e., the mouth) is a surprisingly effective way to infer head movements.
  • HLCC Captures Rich Trajectory Information: The application of Higher-Order Local Cross Correlation proves to be a powerful feature extraction technique for dynamic trajectories, outperforming simpler methods.
  • Integration Potential: The method builds upon existing noise-robust speech recognition systems, suggesting a natural pathway to integrating head gesture recognition with speech recognition without requiring entirely new hardware setups.

Key Results

  • Head Gesture Recognition Accuracy: The proposed acoustic-based method achieved a best accuracy of 90.25% for nineteen kinds of head gestures. This was obtained using HLCC features with a maximum order of 2 and a local pattern length of 8.
  • Comparison with HMM: The proposed method's best score (90.25%) was comparable to, and slightly better than, a Hidden Markov Model (HMM)-based method (best score of 89.48%) that used similar training and testing procedures. This is notable given the proposed method's relative simplicity.
  • Noise Robustness of Localization: The microphone array system demonstrated good noise robustness. Even at low SNRs, the average error distance in localizing utterance positions was less than 2.5 cm, which is considered negligible and comparable to the size of one grid unit in their localization system.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the proposed method is based on the assumption that the user's voice position closely corresponds to their mouth position. While effective, this might have limitations for certain speaking styles or head orientations.
  • The recognition accuracy, while good, is not perfect. There is still room for improvement, especially for more subtle or complex gestures.
  • Future Work:
    • The authors explicitly state their plan to develop a system that integrates head gesture recognition with speech recognition. This would create a more holistic understanding of human communication.
  • Further exploration of different feature extraction techniques and recognition models could potentially improve accuracy.
  • Investigating the system's performance with a wider variety of head gestures and in more diverse acoustic environments would be valuable.

Summary updated

BibTeX
@inproceedings{sasou2010acoustic,
  author = {Akira Sasou and Yasuharu Hashimoto and Katsuhiko Sakaue},
  title = {{Acoustic-based recognition of head gestures accompanying speech}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure