DO as I Mean, Not as I Say: Sequence Loss Training for Spoken Language Understanding

Milind Rao, Pranav Dheram, Gautam Tiwari, Anirudh Raju, Jasha Droppo, Ariya Rastrow, Andreas Stolcke

Abstract

Traditional Spoken Language Understanding (SLU) systems typically rely on a pipeline of Automatic Speech Recognition (ASR) followed by Natural Language Understanding (NLU). This structure often suffers from a "mismatch of objectives," where components are trained on local word accuracy rather than final semantic meaning. We present a Sequence Loss Training approach using the REINFORCE algorithm to directly optimize for semantic metrics like Semantic Error Rate (SemER). Our results demonstrate a 6% relative improvement in performance across large-scale datasets and show that ASR models can be effectively updated using only semantic feedback, enabling robust training even in the absence of human transcripts.

Introduction

In modern voice assistants, the path from audio to action involves two distinct steps: ASR converts speech to text, and NLU extracts intent and slots from that text. These models are traditionally trained using Cross-Entropy (CE) loss, which encourages predicting the next correct word or tag. However, the metrics that define user satisfaction are Word Error Rate (WER) and Semantic Error Rate (SemER).

This pipeline approach creates a bottleneck: ASR errors cascade into the NLU. Because the ASR is unaware of which words are semantically critical, it may perfectly transcribe filler words while failing on a key noun that changes the entire intent. To solve this, we propose a method to train the entire system by "doing as the user means"—optimizing for the final semantic outcome rather than just the intermediate text.

Methodology

The core of our approach is Sequence Loss Training, which utilizes the REINFORCE algorithm (a policy gradient method) to bridge the gap between differentiable training and non-differentiable evaluation metrics.

Joint Modeling and Neural Interfaces

We explore three primary methods for connecting ASR and NLU components to allow for joint optimization:

  1. Compositional (Text): Passing the raw string output, serving as a baseline.
  2. Subword Tokens: Using Gumbel-softmax to pass categorical tokens.
  3. Neural Interface: Passing hidden layer features directly from the ASR decoder to the NLU. This joint approach allows the NLU to process the rich, ambiguous features of the audio signal rather than a single fixed transcript.
Architecture of the joint ASR-NLU system showing different interface types.

Minimum Semantic Error Rate (mSemER)

Instead of minimizing word-level cross-entropy, we define a loss based on the final semantic output (intent + slots). This allows the model to prioritize the "meaning" of the utterance. By using the REINFORCE trick, we can backpropagate through the non-differentiable beam search process used during decoding.

Transcript-Free Training

A significant innovation in this work is the ability to perform training without human-generated transcripts. In production environments, we often have audio and the resulting semantic action (e.g., a user successfully set an alarm) but no word-for-word text. We demonstrate that by treating the ASR's best guess as a "pseudo-transcript" and applying semantic feedback, we can update the ASR model to be more accurate for future downstream tasks. This "weakly supervised" learning allows models to improve directly from real-world user interactions.

Key Results

Our experiments on both public benchmarks and large-scale internal datasets show that semantic optimization consistently outperforms standard training regimes. On the public Fluent Speech dataset, our joint mSLU model achieved a state-of-the-art Interpretation Error Rate (IRER) of 0.39%.

Performance comparison of different training objectives and model architectures on internal datasets.

As shown in the results table above, the Joint mSLU model (Row 4c) provides significant relative error rate reductions (WERR and SemERR) compared to the compositional baseline. Specifically, we observed a 6% relative improvement in both ASR and NLU metrics. Furthermore, in transcript-free scenarios, the models achieved a ~5.8% improvement in SemER using only intent and slot labels as feedback, proving that semantic signals are sufficient for ASR refinement.

Conclusion

By shifting the training objective from "predicting the next word" to "understanding the user's intent," we can build more robust and efficient SLU systems. Sequence Loss Training not only improves accuracy but also opens the door to learning from un-transcribed data, a critical capability for scaling voice AI in the real world. Future work will focus on reducing the computational complexity of beam-search-based training and exploring these methods in open-domain conversational contexts.

Summary updated

BibTeX
@inproceedings{rao2021do,
  author = {Milind Rao and Pranav Dheram and Gautam Tiwari and Anirudh Raju and Jasha Droppo and Ariya Rastrow and Andreas Stolcke},
  title = {{DO as I Mean, Not as I Say: Sequence Loss Training for Spoken Language Understanding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414566},
}
Figure