Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces

Milind Rao, Anirudh Raju, Pranav Dheram, Bach Bui, Ariya Rastrow

Introduction

This research from the Amazon Alexa team proposes a significant shift in voice processing architecture: moving from traditional two-stage cascading systems to an end-to-end "Joint SLU" model. In standard systems, an Automatic Speech Recognition (ASR) component converts audio to text, which is then processed by a Natural Language Understanding (NLU) component to determine intent and extract slots.

This traditional approach suffers from two primary flaws. First, error propagation occurs when ASR mistakes (e.g., hearing "alarm" as "a farm") lead the NLU to process incorrect information. Second, text is a lossy interface that strips away acoustic cues like prosody and tone, as well as the underlying uncertainty or "confusion" the ASR might have regarding specific words. The proposed Joint SLU model addresses these issues by replacing the text interface with a rich neural hidden layer.

Architectural Approaches

The researchers compared three distinct architectural strategies to evaluate the benefits of joint training and neural interfaces:

  1. Direct Audio-to-Intent: A lightweight model that maps audio features directly to an intent label. While efficient for low-resource "edge" devices, it does not provide a transcript or extract specific slots.
  2. Compositional (The Baseline): A standard pipeline where a Listen Attend Spell (LAS) ASR model generates a text transcript, which is then fed into a separate Neural NLU model. These components are trained independently.
  3. Joint SLU (The Innovation): This model is fully differentiable from end to end. Instead of passing plain text, the ASR's decoder hidden states ($h_i$) are concatenated with word embeddings ($e_i$) to create a "neural interface." This allows the NLU to "see" the internal representations and uncertainty of the ASR.
Architecture of the Joint SLU model compared to compositional and direct audio-to-intent approaches.

Datasets and Training

The study utilized massive, in-house far-field acoustic datasets from Amazon to ensure robust results:

  • 13 Intent Data: 1.36 million utterances (1,150 hours) for common tasks.
  • 42 Intent Data: 8 million utterances (6,000 hours) for broader coverage.
  • Clean 18 Intent Dataset: 4 million utterances (2,500 hours) for high-accuracy evaluation.
  • ASR-only Data: 23,000 hours of transcribed speech used to pre-train the ASR component.

Training such a large system required a multi-step approach—pre-training the ASR and NLU components separately before fine-tuning them together—to ensure gradient stability.

Key Insights and Results

The researchers evaluated performance using Word Error Rate (WER) for ASR, and Semantic Error Rate (SemER) and Intent Classification Error Rate (ICER) for NLU.

The Power of the Neural Interface

Passing hidden neural states proved more effective than passing plain text. It allowed the NLU to account for ASR confusion encoded in the hidden layers, leading to a 2.7% improvement in SemER. As shown in the results table, the Joint SLU model (Row 3) begins to approach "Oracle" levels (Row 1), where the NLU is given perfect ground-truth text.

Semantic Feedback for ASR

A surprising finding was that joint training actually improved ASR performance, reducing WER by 3.8% to 6.6%. By receiving feedback from the NLU, the ASR component learns which word sequences make more "semantic sense," helping it resolve ambiguous audio signals.

Performance comparison across different SLU architectures showing metrics for WER, SemER, IRER, and ICER.

Conclusion and Future Work

The Joint SLU model demonstrates that an all-neural interface successfully recovers much of the information lost in traditional text-based pipelines. While the current study focused on "1-best" hypotheses, future work will explore "n-best" interfaces to pass multiple hypotheses through the neural layers. Additionally, while these models excel at specialized voice assistant tasks, scaling them to open-domain conversation remains a significant challenge for the field.

Summary updated

BibTeX
@inproceedings{rao2020speech,
  author = {Milind Rao and Anirudh Raju and Pranav Dheram and Bach Bui and Ariya Rastrow},
  title = {{Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces}},
  booktitle = {Interspeech},
  year = {2020},
  doi = {10.21437/interspeech.2020-2976},
}
Figure