Speech generation from hand gestures based on space mapping

Aki Kunikoshi, Yu Qiao, Nobuaki Minematsu, Keikichi HiroseInterspeech 2009 · DOI

Abstract

Traditional text-to-speech (TTS) systems for individuals with speech disabilities often lack prosody and real-time control. This paper presents a framework that translates hand gestures into audible Japanese vowels in real-time using Gaussian Mixture Models (GMMs). By mapping the topological space of hand positions to the acoustic space of speech, the system enables more expressive and natural communication, allowing users to control articulation and duration directly.

Introduction

Individuals suffering from dysarthria or other speech-related disabilities frequently rely on TTS synthesizers. While functional, these systems are often slow and produce robotic, monotonous output because they require symbolic input (typing). This research explores an alternative: replacing the articulatory gestures of the tongue with the voluntarily movable gestures of the hands. The goal is to create a continuous mapping between hand movements and speech sounds that feels intuitive and maintains high intelligibility.

Methodology

GMM-Based Media Conversion

The core of the proposed system is a technique known as GMM-based media conversion. This method, often used in voice conversion, treats hand gestures and speech sounds as two distinct mathematical "spaces."

The system interface showing the CyberGlove input and the real-time 3D hand model used for gesture capture.

Space Mapping and Structural Representation

The researchers define a mapping function $f(a) = b$, where $a$ represents a hand gesture vector and $b$ represents a speech cepstrum vector. To ensure the mapping is intuitive, the authors utilized a "structural representation" approach. They calculated distance matrices to compare the "shape" of the gesture space against the acoustic relationships found in the standard Japanese vowel chart.

Conceptual diagram of the mapping function f translating a hand motion sequence into the target speech space.

Data Acquisition and Optimization

Data was captured using a "CyberGlove" equipped with 18 sensors to record precise finger and hand positions. Speech was synthesized using the STRAIGHT system, which allows for high-quality manipulation of acoustic parameters. To find the optimal gesture set for the five Japanese vowels (/a/, /i/, /u/, /e/, /o/), the team performed Principal Component Analysis (PCA) on 28 basic gestures, selecting those whose topological arrangement most closely mirrored the acoustic distances between the vowels.

Key Insights

  • Topological Alignment: The most critical finding is that the relationship between gestures must mirror the relationship between sounds. If two vowels are acoustically similar, their corresponding gestures should be proportionally close in the gesture space.
  • Agency in Articulation: Unlike discrete trigger systems, this continuous mapping allows the user to "play" the vocal tract. This provides agency over the "crispness" or "slurredness" of the output, mimicking natural speech transitions.

Results and Discussion

In listening tests with Japanese adults, vowels generated from the optimized gesture sets achieved an impressive intelligibility rate of 99.2% to 99.6%. In contrast, a preliminary gesture set chosen without structural mapping achieved only 95.2%, highlighting the importance of topological matching.

Limitations and Future Work

While successful for vowels, the current system does not yet support consonants or complex syllables. Future research will focus on:

  1. Prosody Control: Integrating control over pitch (F0) and emotional expression.
  2. User Adaptation: Developing automated methods to customize gesture mappings for individuals with varying degrees of hand mobility.
  3. Consonant Synthesis: Expanding the vocabulary beyond basic vowel sounds.

Summary updated

BibTeX
@inproceedings{kunikoshi2009speech,
  author = {Aki Kunikoshi and Yu Qiao and Nobuaki Minematsu and Keikichi Hirose},
  title = {{Speech generation from hand gestures based on space mapping}},
  booktitle = {Interspeech},
  year = {2009},
  doi = {10.21437/interspeech.2009-102},
}
Figure