Robust Understanding of Robot-Directed Speech Commands Using Sequence to Sequence With Noise Injection

Yuuki Tada, Yoshinobu Hagiwara, Hiroki Tanaka, Tadahiro TaniguchiFrontiers Robotics AI 2019 · DOI · OpenReview

Abstract

Service robots operating in domestic environments must reliably interpret spoken commands, yet Automatic Speech Recognition (ASR) systems frequently produce errors due to environmental noise or non-native accents. This paper introduces Seq2Seq-NI, a method that enhances robot understanding by training an encoder-decoder neural network with phoneme-level noise injection. By simulating ASR errors during training, the system learns to map corrupted speech recognition results to accurate robot command sequences, significantly improving robustness over traditional rule-based methods.

Introduction

The standard pipeline for robot speech understanding involves capturing a speech signal, processing it through an ASR system, performing semantic parsing, and finally executing the robot action. However, the transition from ASR to semantic parsing is often a "weak link." Traditional rule-based parsers are brittle; even a single misrecognized phoneme can lead to total command failure. While ASR technology continues to improve, errors remain inevitable, necessitating a more robust approach to semantic understanding.

Overview of the proposed system pipeline, showing the flow from speech signal to inferred command sequences via various semantic parsing methods.

Methodology: Seq2Seq with Noise Injection

The authors propose a semantic parsing system based on a Sequence-to-Sequence (Seq2Seq) architecture equipped with an attention mechanism.

Architecture

The model uses an encoder-decoder structure where the encoder processes input phoneme sequences and the decoder generates robot commands. The attention mechanism allows the decoder to focus on specific parts of the input sequence during generation, which is critical for handling long or complex instructions.

Detailed overview of the Seq2Seq architecture with attention used for mapping phoneme sequences to robot commands.

Noise Injection Strategy

The core innovation is the Stochastic Deformation Model (SDM) used during the training phase. Rather than training on "perfect" text, the authors inject artificial noise into phoneme sequences through three operations:

  • Insertion: Adding random phonemes.
  • Deletion: Removing phonemes.
  • Substitution: Swapping phonemes for others.

This phoneme-level approach is superior to word-level noise because it captures the phonological similarities common in ASR errors (e.g., "take it" becoming "tickets"). By training on this "noisy" data, the neural network learns to recover the intended meaning from garbled input.

Experimental Setup

The system was evaluated using a synthetic dataset of 10,000 training sentences generated via the GPSR Sentence Generator (based on RoboCup@Home tasks). To test real-world performance, four participants recorded test sentences in noisy domestic environments. These recordings were processed through two commercial ASR systems—Google Speech API and CMU Sphinx—to provide a realistic baseline of recognition errors.

Key Results and Insights

The Seq2Seq-NI model significantly outperformed all baselines. In tests using the Google Speech API, which suffered from a near 100% sentence error rate in noisy conditions, traditional rule-based and standard Seq2Seq models failed completely. In contrast, Seq2Seq-NI achieved a 24% success rate in perfect command reconstruction.

Beyond success rates, the model demonstrated remarkable robustness. It successfully inferred commands like Grasp(camera) even when the ASR output was nonsensical (e.g., "next detect camera tickets"). A key finding was that the optimal noise level during training (ρ) directly correlates with the expected ASR error rate during deployment, suggesting that noise injection acts as a specialized form of data augmentation for speech robustness.

Conclusion and Future Work

Seq2Seq-NI provides a robust framework for robot command understanding in the face of inevitable ASR errors. However, the current system operates on a closed vocabulary of 110 lexicons. Future research will focus on handling "Out of Vocabulary" words, implementing on-line learning for new objects, and integrating environmental context—such as visual data—to further improve the robot's interpretive accuracy.

Summary updated

Abstract

This paper describes a new method that enables a service robot to understand spoken commands in a robust manner using off-the-shelf automatic speech recognition (ASR) systems and an encoder-decoder neural network with noise injection. In numerous instances, the understanding of spoken commands in the area of service robotics is modeled as a mapping of speech signals to a sequence of commands that can be understood and performed by a robot. In a conventional approach, speech signals are recognized, and semantic parsing is applied to infer the command sequence from the utterance. However, if errors occur during the process of speech recognition, a conventional semantic parsing method cannot be appropriately applied because most natural language processing methods do not recognize such errors. We propose the use of encoder-decoder neural networks, e.g., sequence to sequence, with noise injection. The noise is injected into phoneme sequences during the training phase of encoder-decoder neural network-based semantic parsing systems. We demonstrate that the use of neural networks with a noise injection can mitigate the negative effects of speech recognition errors in understanding robot-directed speech commands i.e., increase the performance of semantic parsing. We implemented the method and evaluated it using the commands given during a general purpose service robot (GPSR) task, such as a task applied in RoboCup@Home, which is a standard service robot competition for the testin...
BibTeX
@article{tada2019robust,
  author = {Yuuki Tada and Yoshinobu Hagiwara and Hiroki Tanaka and Tadahiro Taniguchi},
  title = {{Robust Understanding of Robot-Directed Speech Commands Using Sequence to Sequence With Noise Injection}},
  journal = {Frontiers Robotics AI},
  year = {2019},
  doi = {10.3389/frobt.2019.00144},
}
Figure