Abstract
Humanoid robots often struggle with speech recognition due to "ego-noise" generated by their own internal motors and fans. This paper proposes a two-stage Deep Neural Network (DNN) architecture that incorporates the robot's motor state into a bottleneck feature extraction process. By creating noise-adaptive representations, the system achieves an 11.5% relative reduction in Phoneme Error Rate (PER) compared to standard DNN baselines.
Introduction
Interaction is a core function of humanoid robots like Pepper or JIBO. However, their microphones are typically located near internal mechanical components, leading to significant ego-noise. This noise is non-stationary, changing dynamically with the robot's movements, which makes traditional spectral subtraction methods ineffective. While previous attempts used motor data as simple auxiliary inputs, they often failed to capture the complex acoustic-mechanical relationships. This research explores using bottleneck features (BNF) to learn more robust, noise-aware representations.
Methodology
The proposed approach utilizes a two-stage training strategy to integrate acoustic features with mechanical state information.
Stage 1: Bottleneck Feature Extraction
The system first processes standard Mel-Frequency Cepstral Coefficients (MFCCs) alongside auxiliary motor state features. The motor state is represented as a binary "on/off" signal, which is one-hot encoded to indicate whether the robot is stationary or moving.
These inputs are fed into a DNN with a narrow "bottleneck" layer. This architecture forces the network to compress high-dimensional data into a low-dimensional, noise-aware vector. The researchers experimented with different training targets for this stage, finding that predicting phonemes (supervised learning) yielded the most effective features for subsequent recognition tasks.
Stage 2: Acoustic Model Training
The compressed features from the bottleneck layer are then passed to a second, larger DNN. This second network uses both the original MFCCs and these new, motor-aware bottleneck features to perform the final task of phoneme recognition.
Experimental Setup
The system was evaluated using the TIMIT speech database mixed with real ego-noise recorded from a JIBO humanoid robot. Two noise conditions were tested: "fan noise" (motor off) and "movement noise" (motor on, specifically head shaking). Evaluation was conducted across various Signal-to-Noise Ratios (SNR) from 5 dB to 20 dB to simulate different environments.
Results and Discussion
The study found that a 40-dimensional bottleneck layer placed in the middle of the network (layer 2 or 3) provided the best balance between performance and computational complexity. The best-performing model, which used phoneme targets for the bottleneck stage (BN-DNN-PHN), achieved an 11.5% relative improvement in Phoneme Error Rate (PER) over a baseline DNN that used no motor data.
Key insights include:
- Supervision is critical: Training the bottleneck to predict speech sounds (phonemes) is more effective than unsupervised reconstruction or predicting the motor state itself.
- Representation over raw data: Forcing motor information through a bottleneck alongside audio allows the network to learn a specific representation of how mechanical states distort audio, rather than just treating the state as a simple flag.
- Consistency: The improvement held steady across both stationary (fan noise) and active movement scenarios.
Conclusion
Augmenting acoustic features with motor-state-dependent bottleneck features significantly enhances speech recognition in noisy robotic environments. This approach outperforms previous methods that used motor data as a simple auxiliary input without compression. Future work will look into more granular motor data—such as joint velocity and multi-motor coordination—to further refine the system's ability to handle complex, real-world movements.
Summary updated