Speech robot mimicking human articulatory motion

Kotaro Fukui, Toshihiro Kusano, Yoshikazu Mukaeda, Yuto Suzuki, Atsuo Takanishi, Masaaki HondaInterspeech 2010

TL;DR

This paper introduces a novel control method for a humanoid talking robot, Waseda Talker No. 7 Refined II, that mimics human articulatory trajectories captured via electromagnetic articulography (EMA). This approach significantly improves speech production speed and smoothness compared to previous phoneme-based concatenation methods, bringing the robot closer to human-like speech capabilities.

What Problem Does This Paper Address?

Creating realistic and natural-sounding speech from robots has been a long-standing challenge. Traditional methods for controlling talking robots often rely on pre-programmed, discrete phoneme configurations. While this allows for speech generation, it leads to jerky, unnatural movements and significantly slower speech production than humans. This is because the robot's motors must execute rapid, discrete movements to transition between phonemes, which is mechanically inefficient and doesn't account for the continuous, fluid nature of human articulation. The authors aim to bridge this gap by developing a control strategy that directly emulates the dynamic, continuous movements of the human speech apparatus.

Methods

The core innovation lies in their data-driven control approach. Instead of relying on pre-defined phoneme parameters, they:

  • Acquire Human Articulatory Data: They use an Electromagnetic Articulography (EMA) system to capture the real-time 3D movement data of human articulators (lips, tongue, jaw, palate). This system uses transmitter and receiver coils to precisely track the position of these points.
  • Convert EMA Data to Robot Control Parameters: This is a multi-step process:
  • Geometric Transformation: The human tongue's size and shape, as captured by EMA, are scaled and adjusted to match the robot's physical dimensions and palate shape.
  • Inverse Kinematics: The transformed EMA data (representing desired positions of key tongue points) is used to calculate the necessary joint angles and movements for the robot's tongue mechanism. This involves solving inverse kinematics for the robot's slider-crank links.
  • Lip and Vocal Cord Control: For lips, the EMA data is used to estimate the open area, which then dictates the robot's lip parameters. For vocal cords, while EMA data doesn't directly provide parameters, simultaneous sound data is used to infer voiced/unvoiced timing for control.
  • Proposed Trajectory Production: The new method generates continuous trajectories for the robot's articulators by directly mapping the human EMA data, leading to smoother, more efficient motor movements.

Datasets

  • Human Speech Data: The paper utilizes EMA data collected from an adult male speaking Japanese phrases, specifically the utterance "/sasisuseso/". This data captures the 2D X-Y components of movement for nine articulation points (upper lip, lower lip, jaw, soft palate, and four points on the tongue).
  • No new benchmarks were introduced. The paper focuses on improving the control of an existing robot by leveraging existing human motion capture technology.

Key Insights and Ideas

  • Human Trajectory Mimicry is Key: The most significant insight is that directly mimicking the trajectory of human articulators, rather than just their static configurations, is crucial for achieving natural and fast speech.
  • EMA as a Powerful Tool: EMA proves to be an effective and practical method for capturing the dynamic movements required for this research, overcoming limitations of X-ray (damage) and MRI (time-consuming).
  • Inverse Kinematics for Complex Mechanisms: The paper demonstrates the successful application of inverse kinematics to translate complex human movements into control signals for the robot's intricate tongue mechanism.
  • Reduced Motor Load and Error: By adopting a smoother, human-like trajectory, the robot's motors experience less strain, and the error between target and actual output is reduced compared to previous methods.

Key Results

  • Human-like Speech Speed: The new control method allows the robot to produce speech at a speed comparable to human speech, significantly faster than previous methods which were up to three times slower (e.g., 150ms transition time for /sasisuseso/ vs. human-like speed).
  • Smoother Motor Movements: Experimental results (shown in Figure 9) indicate that the encoder output for the robot's tongue motor is significantly smoother with the new control method compared to the previous one.
  • Improved Speech Quality: Figure 10 shows that the waveform produced by the robot using the new control method is more natural and less distorted, particularly for consonant sounds.
  • Reduced Motor Load and Error: The authors report reduced load on motors and a smaller error between target and actual output compared to previous methods.

Limitations and Future Work

  • Data Dependency: The proposed method is heavily reliant on the availability of EMA data. If EMA data for a specific speech segment or language is not available, this method cannot be directly applied.
  • Tongue Shape Estimation: Estimating the robot's tongue shape (specifically points q3 and q4) from EMA data was challenging due to the limited number of EMA points and the complexity of the tongue's posterior shape. This required some heuristic estimation.
  • Future Work:
    • Broader Vocabulary: The current success is demonstrated for specific phrases. Further work is needed to extend this to a wider range of words and languages.
  • Muscle Structure Analysis: The authors suggest analyzing the human tongue's muscle structure to develop a simpler, more robust control model that might not solely rely on EMA data.
  • New Robot Mechanism Design: Based on muscle structure analysis, they aim to develop a new robot tongue mechanism that better reflects human biomechanics.

Summary updated

BibTeX
@inproceedings{fukui2010speech,
  author = {Kotaro Fukui and Toshihiro Kusano and Yoshikazu Mukaeda and Yuto Suzuki and Atsuo Takanishi and Masaaki Honda},
  title = {{Speech robot mimicking human articulatory motion}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure