Augmenting Bottleneck Features of Deep Neural Network Employing Motor State for Speech Recognition at Humanoid Robots

Moa Lee, Joon-Hyuk ChangarXiv 2018 · arXiv

Abstract

Humanoid robots often struggle with speech recognition due to "ego-noise" generated by their own internal motors and fans. This paper proposes a two-stage Deep Neural Network (DNN) architecture that incorporates the robot's motor state into a bottleneck feature extraction process. By creating noise-adaptive representations, the system achieves an 11.5% relative reduction in Phoneme Error Rate (PER) compared to standard DNN baselines.

Introduction

Interaction is a core function of humanoid robots like Pepper or JIBO. However, their microphones are typically located near internal mechanical components, leading to significant ego-noise. This noise is non-stationary, changing dynamically with the robot's movements, which makes traditional spectral subtraction methods ineffective. While previous attempts used motor data as simple auxiliary inputs, they often failed to capture the complex acoustic-mechanical relationships. This research explores using bottleneck features (BNF) to learn more robust, noise-aware representations.

Methodology

The proposed approach utilizes a two-stage training strategy to integrate acoustic features with mechanical state information.

Stage 1: Bottleneck Feature Extraction

The system first processes standard Mel-Frequency Cepstral Coefficients (MFCCs) alongside auxiliary motor state features. The motor state is represented as a binary "on/off" signal, which is one-hot encoded to indicate whether the robot is stationary or moving.

Motor state encoding from a head-shaking robot.

These inputs are fed into a DNN with a narrow "bottleneck" layer. This architecture forces the network to compress high-dimensional data into a low-dimensional, noise-aware vector. The researchers experimented with different training targets for this stage, finding that predicting phonemes (supervised learning) yielded the most effective features for subsequent recognition tasks.

Stage 2: Acoustic Model Training

The compressed features from the bottleneck layer are then passed to a second, larger DNN. This second network uses both the original MFCCs and these new, motor-aware bottleneck features to perform the final task of phoneme recognition.

The two-stage process: (1) Extracting motor-state dependent bottleneck features and (2) Training the final acoustic model.

Experimental Setup

The system was evaluated using the TIMIT speech database mixed with real ego-noise recorded from a JIBO humanoid robot. Two noise conditions were tested: "fan noise" (motor off) and "movement noise" (motor on, specifically head shaking). Evaluation was conducted across various Signal-to-Noise Ratios (SNR) from 5 dB to 20 dB to simulate different environments.

Results and Discussion

The study found that a 40-dimensional bottleneck layer placed in the middle of the network (layer 2 or 3) provided the best balance between performance and computational complexity. The best-performing model, which used phoneme targets for the bottleneck stage (BN-DNN-PHN), achieved an 11.5% relative improvement in Phoneme Error Rate (PER) over a baseline DNN that used no motor data.

Key insights include:

  • Supervision is critical: Training the bottleneck to predict speech sounds (phonemes) is more effective than unsupervised reconstruction or predicting the motor state itself.
  • Representation over raw data: Forcing motor information through a bottleneck alongside audio allows the network to learn a specific representation of how mechanical states distort audio, rather than just treating the state as a simple flag.
  • Consistency: The improvement held steady across both stationary (fan noise) and active movement scenarios.

Conclusion

Augmenting acoustic features with motor-state-dependent bottleneck features significantly enhances speech recognition in noisy robotic environments. This approach outperforms previous methods that used motor data as a simple auxiliary input without compression. Future work will look into more granular motor data—such as joint velocity and multi-motor coordination—to further refine the system's ability to handle complex, real-world movements.

Summary updated

Abstract

As for the humanoid robots, the internal noise, which is generated by motors, fans and mechanical components when the robot is moving or shaking its body, severely degrades the performance of the speech recognition accuracy. In this paper, a novel speech recognition system robust to ego-noise for humanoid robots is proposed, in which on/off state of the motor is employed as auxiliary information for finding the relevant input features. For this, we consider the bottleneck features, which have been successfully applied to deep neural network (DNN) based automatic speech recognition (ASR) system. When learning the bottleneck features to catch, we first exploit the motor on/off state data as supplementary information in addition to the acoustic features as the input of the first deep neural network (DNN) for preliminary acoustic modeling. Then, the second DNN for primary acoustic modeling employs both the bottleneck features tossed from the first DNN and the acoustics features. When the proposed method is evaluated in terms of phoneme error rate (PER) on TIMIT database, the experimental results show that achieve obvious improvement (11% relative) is achieved by our algorithm over the conventional systems.
BibTeX
@misc{lee2018augmenting,
  author = {Moa Lee and Joon-Hyuk Chang},
  title = {{Augmenting Bottleneck Features of Deep Neural Network Employing Motor State for Speech Recognition at Humanoid Robots}},
  year = {2018},
  eprint = {1808.08702},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.08702},
}
Figure