Abstract
Inspired by the behavior of humans talking in noisy environments, we propose an embodied embedded cognition approach to improve automatic speech recognition (ASR) systems for robots in challenging environments, such as with ego noise, using binaural sound source localization (SSL). The approach is verified by measuring the impact of SSL with a humanoid robot head on the performance of an ASR system. More specifically, a robot orients itself toward the angle where the signal-to-noise ratio (SNR) of speech is maximized for one microphone before doing an ASR task. First, a spiking neural network inspired by the midbrain auditory system based on our previous work is applied to calculate the sound signal angle. Then, a feedforward neural network is used to handle high levels of ego noise and reverberation in the signal. Finally, the sound signal is fed into an ASR system. For ASR, we use a system developed by our group and compare its performance with and without the support from SSL. We test our SSL and ASR systems on two humanoid platforms with different structural and material properties. With our approach we halve the sentence error rate with respect to the common downmixing of both channels. Surprisingly, the ASR performance is more than two times better when the angle between the humanoid head and the sound source allows sound waves to be reflected most intensely from the pinna to the ear microphone, rather than when sound waves arrive perpendicularly to the membrane.
Introduction
Automatic Speech Recognition (ASR) systems often struggle in real-world robotic applications due to "ego noise"—the internal acoustic interference from cooling fans and motors—and environmental reverberation. While humans naturally navigate the "cocktail party effect" by focusing on specific sound sources, robots often treat audio as a flat data stream, losing critical spatial information. Traditional methods often rely on large microphone arrays or simple "downmixing" (averaging left and right channels), which can inadvertently destroy spatial cues and lower the Signal-to-Noise Ratio (SNR).
This research explores embodied cognition, positing that a robot’s physical structure and its ability to move are active tools for improving perception. By mimicking human auditory anatomy and behavior, robots can significantly enhance their speech recognition accuracy in noisy environments.
Biomimetic Auditory Processing
The proposed system utilizes a multi-stage, bio-inspired pipeline to process sound similarly to the mammalian auditory pathway.
Peripheral and Midbrain Modeling
Sound signals are first processed through a Gammatone filterbank, which emulates the human cochlea by decomposing audio into frequency components. These components are then fed into a Spiking Neural Network (SNN) that models key areas of the midbrain:
- Medial Superior Olive (MSO): Specialized in extracting Interaural Time Differences (ITDs), which are most effective for low-frequency sounds.
- Lateral Superior Olive (LSO): Specialized in extracting Interaural Level Differences (ILDs), which are more pronounced at high frequencies due to the head shadow effect.
The Inferior Colliculus (IC) then integrates these cues using Bayesian inference to create a coherent spatial representation of the sound source.
System Architecture
The overall architecture integrates these bio-inspired components with modern machine learning. A feedforward Multilayer Perceptron (MLP) handles non-linearities caused by the robot's specific head geometry and intense ego noise. The final localized angle is used to orient the robot, and the optimized audio signal is passed to the DOCKS system for speech recognition.
Key Insights: The Power of Embodiment
A central finding of this study is the "Pinna Advantage." Contrary to intuition, ASR performance does not peak when the sound source is directly in front of the robot. Instead, it is maximized at specific angles (approximately 45° and 150°) where sound waves reflect most intensely off the pinna (the outer ear structure) into the microphone.
By localizing a sound and rotating its head to these optimal angles, the robot physically increases the SNR of the speech signal. The study also confirms that selecting the "cleaner" ear's monaural signal is far more effective than standard dual-channel averaging, which often introduces destructive interference.
Results and Conclusion
The system was tested using the TIMIT Core-Test-Set in a virtual reality setup with 13 loudspeakers. Key results include:
- Error Reduction: Orienting the robot to the optimal angle and selecting the best microphone channel halved the sentence error rate (SER) compared to standard averaging.
- Efficiency: The dynamic localization system converged on the correct angle in three iterations or fewer, enabling real-time interaction.
- Robustness: The system remained highly accurate even in the presence of ~60 dB of internal ego noise from the iCub robot's cooling system.
Future work will focus on increasing localization granularity beyond the current 15° bins and integrating auditory models with computer vision for "audio-visual spatial attention," helping robots disambiguate sources in even more complex environments.
Summary updated