Improving End-to-end Models for Set Prediction in Spoken Language Understanding

Hong-Kwang Jeff Kuo, Zoltán Tüske, Samuel Thomas, Brian Kingsbury, George Saon

TL;DR

This paper tackles the challenge of training Spoken Language Understanding (SLU) models when verbatim transcripts are unavailable and only unordered "sets" of semantic entities are provided. By introducing a novel data augmentation technique and an implicit attention-based alignment method to recover spoken order, the researchers achieved an 11.4% F1 score improvement for RNN-Transducer models, effectively matching the performance of models trained on expensive, full verbatim transcripts.

What Problem Does This Paper Address?

Traditional SLU systems are often "cascaded": an Automatic Speech Recognition (ASR) system transcribes audio to text, and a Natural Language Understanding (NLU) system extracts meaning. End-to-End (E2E) models simplify this by mapping audio directly to semantic entities (e.g., "DALLAS" as a destination).

However, training E2E models usually requires verbatim transcripts, which are expensive to produce (taking 5–10x real-time for a human). It is much cheaper to collect "sets" of entities (e.g., from database logs or form-fills), but these sets often lack the original spoken order. Popular E2E architectures like RNN-Transducers (RNN-T) are inherently monotonic—they expect to predict labels in the order they appear in the audio. When forced to learn entities in an arbitrary (e.g., alphabetical) order, performance plummets.

Methods

The authors propose two primary strategies to bridge the gap between unordered entity sets and the sequential nature of speech:

  1. Set-Based Data Augmentation: During a pre-training phase, the model is exposed to ground-truth entities in various random orderings. This "shuffling" helps the model become more robust to order mismatches before it is fine-tuned on a standardized (alphabetical) order.
  2. Reordering via Alignment: The authors propose two ways to "guess" the original spoken order of a set of entities:
  3. Explicit Keyword Search: Using a hybrid ASR model to find where specific entity keywords occur in the audio.
  4. Implicit Attention Alignment: Using the attention weights from an encoder-decoder model to calculate the "average time position" ($t_i$) for each entity. By sorting entities based on these time-stamps, the training data is reorganized into "natural spoken order."

The study evaluates these techniques across three architectures: RNN-Transducers (RNN-T), Attention-based LSTM encoder-decoders, and Conformer encoder-decoders.

Datasets

  • ATIS (Air Travel Information Systems): A standard SLU benchmark containing queries about flights.
  • Noisy ATIS: A custom version created by the authors by adding street noise (5-15 dB SNR) to simulate real-world conditions.
  • Switchboard-300: Used for initial ASR pre-training of the models.

Key Insights and Ideas

  • The Monotonicity Trap: RNN-T models are particularly sensitive to order. When trained on alphabetical entities instead of spoken order, their F1 score dropped by nearly 12%. Attention-based models are more flexible but still saw a ~2% drop.
  • Attention as a Compass: The "aha moment" is that even a model trained on unordered data develops "sharp focus" in its attention layers. By simply looking at where the model thinks a word is in time, we can reorder the training labels to match the audio, creating a virtuous cycle of improvement.
  • Transcripts are Optional: The most significant finding is that with proper reordering, an E2E model trained only on entities can perform as well as one trained on full verbatim transcripts. This suggests a massive potential for cost savings in data collection.

Key Results

  • RNN-T Recovery: On clean data, the RNN-T model's F1 score improved from 81.2% (alphabetical order) to 92.6% (using reordering and augmentation), nearly matching the 93.3% achieved with full transcripts.
  • Conformer Excellence: The Conformer-based attention model reached an F1 of 94.3%, effectively identical to the full-transcript baseline.
  • Noise Robustness: In noisy conditions, the proposed reordering methods were even more critical. The RNN-T model saw an 18.8% absolute increase in F1 score (from 70.7% to 89.5%) when using the proposed techniques.
  • Alignment Accuracy: The implicit attention-based alignment was highly accurate, resulting in only a 2% alignment error, which outperformed the hybrid ASR keyword search method (6% error) on noisy data.

Limitations and Future Work

  • Entity-Only WER: The paper notes that models trained only on entities have a very high Word Error Rate (WER) because they don't output "filler" words. While this doesn't hurt the F1 score (meaning), it might make the model less useful for applications where a partial transcript is still desired for user feedback.
  • Heuristic Dependency: The attention-based reordering relies on a heuristic (average time position). While effective here, it remains to be seen how this scales to much longer utterances or more complex semantic structures where entities might overlap or repeat.
  • Future Direction: The authors suggest that this approach could be extended to automatically collect training data from live conversational systems without any human intervention.

Summary updated

Abstract

The goal of spoken language understanding (SLU) systems is to determine the meaning of the input speech signal, unlike speech recognition which aims to produce verbatim transcripts. Advances in end-to-end (E2E) speech modeling have made it possible to train solely on semantic entities, which are far cheaper to collect than verbatim transcripts. We focus on this set prediction problem, where entity order is unspecified. Using two classes of E2E models, RNN transducers and attention based encoder-decoders, we show that these models work best when the training entity sequence is arranged in spoken order. To improve E2E SLU models when entity spoken order is unknown, we propose a novel data augmentation technique along with an implicit attention based alignment method to infer the spoken order. F1 scores significantly increased by more than 11% for RNN-T and about 2% for attention based encoder-decoder SLU models, outperforming previously reported results.
BibTeX
@inproceedings{kuo2022improving,
  author = {Hong-Kwang Jeff Kuo and Zoltán Tüske and Samuel Thomas and Brian Kingsbury and George Saon},
  title = {{Improving End-to-end Models for Set Prediction in Spoken Language Understanding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=P54DUpqa7K},
}

Loading related content...

Figure