Two-layered audio-visual integration in voice activity detection and automatic speech recognition for robots

Takami Yoshida, Kazuhiro NakadaiInterspeech 2010

TL;DR

This paper introduces a novel "two-layered" audio-visual (AV) integration approach for robots to improve speech recognition in noisy, dynamic environments. By applying AV integration to both voice activity detection (VAD) and the speech recognition decoder, the system achieves more robust performance, especially when audio or visual information is degraded.

What Problem Does This Paper Address?

Human-robot interaction relies heavily on speech recognition (ASR). However, robots often operate in noisy environments (e.g., home, factory floors) where background noise, other speakers, and the robot's own operations can severely degrade ASR performance. Traditional ASR systems struggle with these dynamic conditions. While Audio-Visual (AV) integration, which combines audio and visual cues (like lip movements), has shown promise for improving ASR robustness, most existing AV-ASR systems only apply this integration to the ASR decoding stage. This overlooks the fact that accurate Voice Activity Detection (VAD) is a crucial prerequisite for effective ASR, and VAD itself can also be hampered by noise. Furthermore, real-world robot scenarios often involve imperfect visual data (low resolution, occlusions, changing angles), which many previous AV-ASR studies didn't adequately address.

Methods

The core contribution is a two-layered AV integration framework:

  1. Layer 1: AV-Integrated Voice Activity Detection (AV-VAD):
    • Instead of relying solely on audio or visual cues for VAD, this layer uses a Bayesian network to fuse multiple features.
  2. Inputs include:
  3. Log-likelihood score from an ASR decoder's silence model (Xdvad).
  4. Lip shape features (height and width) derived from visual tracking (Xlip).
  5. Face detection confidence (Xface).
  6. This approach allows for graceful handling of ambiguities and errors in individual features.
  7. Post-processing includes dilation and erosion to refine the detected voice activity segments, followed by temporal margin addition to align with ASR decoder expectations.
  8. Layer 2: AV-Integrated Automatic Speech Recognition Decoder (AV-DEC):
    • This layer integrates audio and visual features directly into the ASR decoding process.
  9. The paper employs an early integration approach, combining features at a lower level.
  10. A key technique is stream weight optimization. This dynamically adjusts the influence of audio and visual features based on their reliability. The weights are learned through linear regression based on Signal-to-Noise Ratio (SNR) and lip image resolution.
  11. The likelihood of an AV feature for a word w at time t is calculated as: bw(xav(t)) = ca * baw(x(t)) + (1 - ca) * bvw(xv(t)), where ca is the audio stream weight.

The overall system architecture combines these two layers with separate audio and visual feature extraction modules.

Datasets

  • Training Data: Acoustically and visually clean Japanese AV speech data from 10 male speakers, consisting of 216 phonemically-balanced words.
  • Evaluation Data:
    • A set of 50 Japanese words not included in the training data.
  • AV data captured by an actual robot, featuring dynamic conditions.
  • Data Characteristics: Audio sampled at 16 kHz, 16 bits. Visual data was 8-bit monochrome, 640x480 pixels, at 33 Hz. For evaluation, the dataset was manipulated to simulate various noise conditions (SNR from 15 dB to -5 dB) and visual degradations (resolutions of one-third and one-fifth of the original).

Key Insights and Ideas

  • Two-Layered Integration is Crucial: Applying AV integration to both VAD and ASR decoding is more effective than focusing on just one stage. This addresses the cascading effect of errors: poor VAD leads to poor ASR, and vice-versa.
  • Bayesian Networks for Robust VAD: Using a Bayesian network for AV-VAD allows for principled fusion of potentially noisy and ambiguous audio and visual cues, leading to more reliable voice activity detection.
  • Dynamic Stream Weighting for ASR: The stream weight optimization, informed by SNR and image resolution, is vital for balancing audio and visual contributions in the ASR decoder. This allows the system to rely more on the reliable modality when the other is degraded.
  • Handling Real-World Visual Imperfections: The evaluation explicitly uses degraded visual data (low resolution), which is a significant step towards practical robot applications, as opposed to relying on ideal lip images.

Key Results

  • VAD Performance (Ex. 1): The AV-VAD system showed improved robustness, particularly when audio SNR was degraded (10 to 5 dB) and visual resolution was reduced (middle or low). On average, improvements of 11.5 percentage points were observed in these challenging conditions compared to audio-only VAD.
  • ASR Performance (Ex. 2): The two-layered AV-ASR system demonstrated better Word Correct Rates (WCRs) than audio-only or visual-only ASR, especially in moderate to high noise conditions (SNR 10, 5, 0 dB). Even at 15 dB SNR, AV integration improved WCR when high or middle resolution images were available.
  • Effect of Margin Addition: The paper notes that WCRs were higher than WDRs, suggesting that the ASR decoder could sometimes correct errors introduced by VAD, partly due to the temporal margin addition technique.

Limitations and Future Work

  • Limitations:
    • The Bayesian network for AV-VAD was trained on clean data, which can lead to performance degradation in some AV integration scenarios where the network hasn't seen such conditions. Multi-condition training is suggested as a solution.
  • The stream weight optimization relies on SNR estimation, which can be difficult in real-world scenarios.
  • Future Work:
    • Address dynamic changes in facial orientation, sound source location, and the number of sound sources.
  • Develop stream weight optimization methods that do not require explicit SNR estimation.

Summary updated

BibTeX
@inproceedings{yoshida2010two,
  author = {Takami Yoshida and Kazuhiro Nakadai},
  title = {{Two-layered audio-visual integration in voice activity detection and automatic speech recognition for robots}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure