Abstract
Traditional text-to-speech (TTS) systems for individuals with speech disabilities often lack prosody and real-time control. This paper presents a framework that translates hand gestures into audible Japanese vowels in real-time using Gaussian Mixture Models (GMMs). By mapping the topological space of hand positions to the acoustic space of speech, the system enables more expressive and natural communication, allowing users to control articulation and duration directly.
Introduction
Individuals suffering from dysarthria or other speech-related disabilities frequently rely on TTS synthesizers. While functional, these systems are often slow and produce robotic, monotonous output because they require symbolic input (typing). This research explores an alternative: replacing the articulatory gestures of the tongue with the voluntarily movable gestures of the hands. The goal is to create a continuous mapping between hand movements and speech sounds that feels intuitive and maintains high intelligibility.
Methodology
GMM-Based Media Conversion
The core of the proposed system is a technique known as GMM-based media conversion. This method, often used in voice conversion, treats hand gestures and speech sounds as two distinct mathematical "spaces."
Space Mapping and Structural Representation
The researchers define a mapping function $f(a) = b$, where $a$ represents a hand gesture vector and $b$ represents a speech cepstrum vector. To ensure the mapping is intuitive, the authors utilized a "structural representation" approach. They calculated distance matrices to compare the "shape" of the gesture space against the acoustic relationships found in the standard Japanese vowel chart.
Data Acquisition and Optimization
Data was captured using a "CyberGlove" equipped with 18 sensors to record precise finger and hand positions. Speech was synthesized using the STRAIGHT system, which allows for high-quality manipulation of acoustic parameters. To find the optimal gesture set for the five Japanese vowels (/a/, /i/, /u/, /e/, /o/), the team performed Principal Component Analysis (PCA) on 28 basic gestures, selecting those whose topological arrangement most closely mirrored the acoustic distances between the vowels.
Key Insights
- Topological Alignment: The most critical finding is that the relationship between gestures must mirror the relationship between sounds. If two vowels are acoustically similar, their corresponding gestures should be proportionally close in the gesture space.
- Agency in Articulation: Unlike discrete trigger systems, this continuous mapping allows the user to "play" the vocal tract. This provides agency over the "crispness" or "slurredness" of the output, mimicking natural speech transitions.
Results and Discussion
In listening tests with Japanese adults, vowels generated from the optimized gesture sets achieved an impressive intelligibility rate of 99.2% to 99.6%. In contrast, a preliminary gesture set chosen without structural mapping achieved only 95.2%, highlighting the importance of topological matching.
Limitations and Future Work
While successful for vowels, the current system does not yet support consonants or complex syllables. Future research will focus on:
- Prosody Control: Integrating control over pitch (F0) and emotional expression.
- User Adaptation: Developing automated methods to customize gesture mappings for individuals with varying degrees of hand mobility.
- Consonant Synthesis: Expanding the vocabulary beyond basic vowel sounds.
Summary updated