Virtual speech reading support for hard of hearing in a domestic multi-media setting

Samer Al Moubayed, Jonas Beskow, Ann-Marie Öster, Giampiero Salvi, Björn Granström, Nic van Son, Ellen OrmelInterspeech 2009

TL;DR

This paper explores the SynFace animated talking head technology as a visual aid for hearing-impaired individuals in home multimedia settings. While preliminary results show limited average gains in speech reception thresholds and effort reduction, the study highlights significant inter-subject variability, suggesting SynFace can be highly beneficial for specific users, particularly in noisy environments like speech with babble.

What Problem Does This Paper Address?

The increasing prevalence of hearing impairment, especially among the elderly, poses a significant challenge to social participation and communication. Traditional assistive devices often fall short in complex home environments that go beyond simple telephone conversations. This paper addresses the need for advanced communication support that integrates with modern multimedia systems (like TVs and home automation) and caters to the diverse needs of hearing-impaired individuals. Specifically, it investigates how a visual lip-reading support system, SynFace, can enhance speech intelligibility and reduce listening effort in various noisy and multi-lingual home scenarios, moving beyond its initial telephony-focused design.

Methods

The core of the SynFace system is a real-time phoneme recognition engine based on a hybrid of recurrent artificial neural networks (ANNs) and hidden Markov models (HMMs). This engine translates speech signals into articulatory information that drives a 3D animated talking face.

Key methodological developments and evaluations include:

  • Wideband Audio Adaptation: SynFace was retrained on wideband (16 kHz) speech data to improve performance with higher quality audio sources beyond traditional telephone bandwidth.
  • Noisy Speech Robustness: The system was evaluated on its ability to handle various noise types (stationary, stereo babble, traffic, action movie) across a range of Signal-to-Noise Ratios (SNRs).
  • Noise Adaptation: A novel approach involved training the phoneme recognizer on a small amount of noisy speech data, demonstrating that this adaptation can generalize and improve performance across a wider range of SNRs.
  • Multilinguality and Language Mapping: To reduce the cost of supporting new languages, researchers explored cross-language evaluation, investigating methods to map phoneme models from one language to another, particularly focusing on mapping English, Flemish, and German models to Swedish. A regression-based approach showed promising accuracy.
  • User Studies: Large-scale user studies were conducted with hearing-impaired individuals across three languages (Swedish, German, Flemish) and varying levels of hearing loss. These studies measured Speech Reception Threshold (SRT) with and without SynFace, and assessed listening effort using a subjective scaling method.

Datasets

  • SpeechDat corpora: Used for initial multilingual training of the SynFace phoneme recognizer.
  • SpeeCon corpus: Used for training the wideband phoneme recognizer (specifically office settings of Swedish speech) and for cross-language mapping experiments (30 minutes of transcribed speech).
  • Custom Stimuli: Word lists were used for SRT measurements in the user studies.

Key Insights and Ideas

  • Inter-Subject Variability is Crucial: The most striking insight is that while average SRT gains might be small, individual users can experience significant benefits from SynFace, especially in challenging conditions like speech with babble noise. This suggests a personalized approach to assistive technology is vital.
  • Wideband Audio Matters: Retraining SynFace on wideband audio significantly improved its phoneme recognition accuracy, indicating the importance of adapting to higher fidelity audio sources common in home multimedia.
  • Noise Adaptation Shows Promise: Even limited adaptation to noisy speech can lead to generalized improvements in phoneme recognition across various SNRs, suggesting a path towards more robust visual speech support in noisy environments.
  • Cross-Language Mapping is Feasible: The research demonstrates that it's possible to achieve reasonable performance for a new language by mapping existing models, significantly reducing the data and time required for new language support.
  • Longer Tests Aren't Always Better: Preliminary analysis of SRT measurements suggested that very long test sessions might lead to listener fatigue, potentially increasing the required SNR for understanding, a critical consideration for user study design.

Key Results

  • Wideband Recognizer: Achieved a 68% correct phoneme frame rate on wideband speech, a significant improvement over previous versions.
  • Noise Adaptation: Adaptation to traffic noise showed a generalized improvement in correct phoneme frame rate across all tested SNR levels.
  • Cross-Language Mapping: Mapping German recognizer output to Swedish achieved 45% accuracy, close to the original Swedish trained recognizer's 52% accuracy.
  • User Studies (Preliminary):
    • Average SRT did not show a significant decrease when using SynFace compared to audio-only conditions.
  • However, a notable benefit was observed for the Swedish participants in speech-in-babble-noise conditions.
  • Effort scaling tests showed no significant difference on average, but again, high inter-subject variability was observed.

Limitations and Future Work

  • Limited Average Benefit: The preliminary analysis of user studies indicates that SynFace, on average, did not provide a consistent, significant benefit for hearing-impaired subjects in terms of SRT or effort reduction.
  • Noise Degradation: SynFace performance is severely degraded by noise, although adaptation shows potential for improvement.
  • Future Work:
    • Further analysis of individual user results to identify characteristics of users who consistently benefit from SynFace.
  • Investigating correlations between different test results for individual subjects.
  • Including additional measurements, such as cognitive tasks related to speech reading, to better understand user characteristics.
  • Exploring the potential of SynFace to leverage broadband audio and adapt to various acoustic signal conditions beyond telephone speech.

Summary updated

BibTeX
@inproceedings{moubayed2009virtual,
  author = {Samer Al Moubayed and Jonas Beskow and Ann-Marie Öster and Giampiero Salvi and Björn Granström and Nic van Son and Ellen Ormel},
  title = {{Virtual speech reading support for hard of hearing in a domestic multi-media setting}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure