Enhanced Robot Speech Recognition Using Biomimetic Binaural Sound Source Localization

Jorge Dávila-Chacón, Jindong Liu, Stefan Wermter

TL;DR

This paper demonstrates that robots can significantly improve their speech recognition accuracy by mimicking human auditory anatomy and behavior. By using a bio-inspired spiking neural network to localize sound and then physically orienting toward the speaker to maximize the "pinna effect" (reflections from the outer ear), the researchers halved the sentence error rate in noisy environments.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often fail in real-world robotic applications due to "ego noise"—the internal hum of cooling fans and motors—and room reverberation. While humans excel at the "cocktail party effect" (focusing on one voice in a noisy room), robots typically struggle because they treat audio as a flat data stream.

Traditional approaches often use large microphone arrays or simple "downmixing" (averaging left and right channels), which can actually destroy spatial cues and lower the Signal-to-Noise Ratio (SNR). This paper explores embodied cognition: the idea that a robot’s physical structure (its "body") and its ability to move are active tools for improving perception.

Methods

The researchers developed a multi-stage, bio-inspired pipeline:

  1. Peripheral Processing: A Gammatone filterbank emulates the human cochlea, decomposing sound into frequency components.
  2. Midbrain Modeling: They implemented a Spiking Neural Network (SNN) that models three key areas of the mammalian auditory pathway:
  3. Medial Superior Olive (MSO): Extracts Interaural Time Differences (ITDs), effective for low frequencies.
  4. Lateral Superior Olive (LSO): Extracts Interaural Level Differences (ILDs), effective for high frequencies.
  5. Inferior Colliculus (IC): Integrates these cues using Bayesian inference to create a coherent spatial representation.
  6. Non-linear Classification: A feedforward Multilayer Perceptron (MLP) handles the remaining non-linearities caused by the robot's specific head geometry and intense ego noise.
  7. ASR Integration (DOCKS): The robot uses the DOCKS system, which combines a domain-independent engine (Google ASR) with a domain-dependent phonetic post-processor to correct errors based on a known vocabulary.

Datasets

  • TIMIT Core-Test-Set: A standard acoustic-phonetic speech corpus consisting of 192 sentences from 24 different speakers.
  • Custom VR Setup: Evaluation was performed in an audio-visual virtual reality setup where 13 loudspeakers were positioned around the robot to simulate different sound source angles in a controlled acoustic environment.

Key Insights and Ideas

  • The Pinna Advantage: One of the most surprising findings was that ASR performance is not best when the sound source is directly in front of the robot (0°) or perpendicular to the ear (90°). Instead, it peaks at angles (around 45° and 150°) where sound waves reflect most intensely off the pinna (the outer ear structure) into the microphone.
  • Move to Hear: The study validates a "perception-action loop." By localizing a sound and then rotating the head to an optimal angle, the robot physically increases the SNR of the speech signal before it even reaches the ASR software.
  • Channel Selection vs. Averaging: The authors show that "downmixing" (averaging L and R channels) is often detrimental. It is far more effective to use SSL to identify the "cleaner" ear and use that monaural signal for recognition.
  • Embodiment Matters: Comparing the iCub (plastic head with noisy fans) to the Soundman (silent wooden dummy head) revealed that while ego noise is a hurdle, the structural benefits of a humanoid head (shadowing and reflection) are consistent across different materials.

Key Results

  • Error Reduction: By orienting the robot to the optimal angle and selecting the best microphone channel, the researchers halved the sentence error rate (SER) compared to standard dual-channel averaging.
  • Localization Speed: The dynamic SSL system was able to converge on the correct sound source angle in 3 iterations or fewer, making it viable for real-time interaction.
  • Robustness: The system maintained high accuracy even with ~60 dB of internal ego noise produced by the iCub’s cooling system.

Limitations and Future Work

  • Granularity: The current system uses 15° localization bins. The authors suggest that moving to a continuous regression model in the final neural layer could allow for 1° resolution.
  • Static vs. Moving Sources: While the system tracks sources quickly, the experiments primarily focused on static speakers. Future work could involve tracking moving targets in more complex "cocktail party" scenarios.
  • Multimodal Integration: The authors propose integrating this auditory model with computer vision, using "audio-visual spatial attention" to help the robot disambiguate sound sources in cluttered environments.

Summary updated

Abstract

Inspired by the behavior of humans talking in noisy environments, we propose an embodied embedded cognition approach to improve automatic speech recognition (ASR) systems for robots in challenging environments, such as with ego noise, using binaural sound source localization (SSL). The approach is verified by measuring the impact of SSL with a humanoid robot head on the performance of an ASR system. More specifically, a robot orients itself toward the angle where the signal-to-noise ratio (SNR) of speech is maximized for one microphone before doing an ASR task. First, a spiking neural network inspired by the midbrain auditory system based on our previous work is applied to calculate the sound signal angle. Then, a feedforward neural network is used to handle high levels of ego noise and reverberation in the signal. Finally, the sound signal is fed into an ASR system. For ASR, we use a system developed by our group and compare its performance with and without the support from SSL. We test our SSL and ASR systems on two humanoid platforms with different structural and material properties. With our approach we halve the sentence error rate with respect to the common downmixing of both channels. Surprisingly, the ASR performance is more than two times better when the angle between the humanoid head and the sound source allows sound waves to be reflected most intensely from the pinna to the ear microphone, rather than when sound waves arrive perpendicularly to the membrane.
BibTeX
@article{davilachacon2019enhanced,
  author = {Jorge Dávila-Chacón and Jindong Liu and Stefan Wermter},
  title = {{Enhanced Robot Speech Recognition Using Biomimetic Binaural Sound Source Localization}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2019},
  url = {https://openreview.net/forum?id=IlY0rHP3sj},
}

Loading related content...

Figure