Improving End-to-End Models for Set Prediction in Spoken Language Understanding

Hong-Kwang Jeff Kuo, Zoltán Tüske, Samuel Thomas, Brian Kingsbury, George Saon

Abstract

The goal of spoken language understanding (SLU) systems is to determine the meaning of the input speech signal, unlike speech recognition which aims to produce verbatim transcripts. Advances in end-to-end (E2E) speech modeling have made it possible to train solely on semantic entities, which are far cheaper to collect than verbatim transcripts. We focus on this set prediction problem, where entity order is unspecified. Using two classes of E2E models, RNN transducers and attention-based encoder-decoders, we show that these models work best when the training entity sequence is arranged in spoken order. To improve E2E SLU models when entity spoken order is unknown, we propose a novel data augmentation technique along with an implicit attention-based alignment method to infer the spoken order. F1 scores significantly increased by more than 11% for RNN-T and about 2% for attention-based encoder-decoder SLU models, outperforming previously reported results.

Introduction

Traditional Spoken Language Understanding (SLU) systems typically follow a "cascaded" architecture: an Automatic Speech Recognition (ASR) component converts audio to text, followed by a Natural Language Understanding (NLU) component that extracts semantic meaning. While effective, this approach suffers from error propagation and high computational overhead. End-to-End (E2E) models offer a streamlined alternative by mapping audio directly to semantic entities (e.g., mapping a spoken city name to a "destination" tag).

A major bottleneck in training E2E models is the requirement for verbatim transcripts, which are expensive and time-consuming to produce. In contrast, "sets" of semantic entities are often readily available from database logs or form-fill data. However, these sets lack the original spoken order. This poses a significant challenge for popular E2E architectures like RNN-Transducers (RNN-T), which are inherently monotonic and expect labels to appear in the order they were spoken.

Proposed Methodology

To bridge the gap between unordered entity sets and sequential speech models, this research introduces two primary strategies to recover or simulate the natural spoken order of entities.

1. Set-Based Data Augmentation

During a pre-training phase, the model is exposed to ground-truth entities in various random orderings. This shuffling process encourages the model to become robust to order mismatches, preventing it from overfitting to a specific arbitrary sequence (like alphabetical order) before it is fine-tuned on standardized data.

2. Reordering via Alignment

The authors propose two methods to infer the original spoken order from an unordered set:

  • Explicit Keyword Search: A hybrid ASR model is utilized to locate specific entity keywords within the audio signal.
  • Implicit Attention Alignment: For encoder-decoder models, the system leverages attention weights to calculate the "average time position" ($t_i$) for each entity. By sorting entities based on these inferred timestamps, the training data is reorganized into a "natural spoken order."
System overview for training E2E SLU models using unordered entity sets and proposed reordering techniques.

Experimental Results

The proposed methods were evaluated using the ATIS (Air Travel Information Systems) benchmark and a custom "Noisy ATIS" dataset (5-15 dB SNR) to simulate real-world environments. The study compared three architectures: RNN-Transducers (RNN-T), Attention-based LSTM encoder-decoders, and Conformer encoder-decoders.

Performance Recovery

The results demonstrate that the "monotonicity trap" is a significant hurdle for RNN-T models. When trained on alphabetical entities, the RNN-T F1 score dropped to 81.2%. However, by applying the proposed reordering and augmentation techniques, the score improved to 92.6%, nearly matching the 93.3% achieved with expensive full transcripts.

Extracted table from page 4 of Improving End-to-End Models for Set Prediction in Spoken Language Understanding

Key Insights

  • Conformer Excellence: The Conformer-based attention model achieved an F1 of 94.3%, effectively reaching parity with the full-transcript baseline. This suggests that with proper alignment, transcripts are essentially optional for high-quality SLU.
  • Noise Robustness: The reordering methods proved even more critical in noisy conditions. The RNN-T model saw an 18.8% absolute increase in F1 score (from 70.7% to 89.5%) when using the proposed techniques.
  • Alignment Accuracy: The implicit attention-based alignment was remarkably accurate, showing only a 2% alignment error, which outperformed the hybrid ASR keyword search method (6% error) on noisy data.

Conclusion

This work demonstrates that E2E SLU models can be trained effectively using only semantic entity sets, provided that the underlying spoken order is recovered. By utilizing attention-based alignment and random order augmentation, researchers can achieve performance levels comparable to models trained on full verbatim transcripts. This finding offers a path toward significantly reducing the cost of data collection for speech-enabled systems and potentially enables the automatic harvesting of training data from live conversational logs.

Summary updated

Abstract

The goal of spoken language understanding (SLU) systems is to determine the meaning of the input speech signal, unlike speech recognition which aims to produce verbatim transcripts. Advances in end-to-end (E2E) speech modeling have made it possible to train solely on semantic entities, which are far cheaper to collect than verbatim transcripts. We focus on this set prediction problem, where entity order is unspecified. Using two classes of E2E models, RNN transducers and attention based encoder-decoders, we show that these models work best when the training entity sequence is arranged in spoken order. To improve E2E SLU models when entity spoken order is unknown, we propose a novel data augmentation technique along with an implicit attention based alignment method to infer the spoken order. F1 scores significantly increased by more than 11% for RNN-T and about 2% for attention based encoder-decoder SLU models, outperforming previously reported results.
BibTeX
@inproceedings{kuo2022improving,
  author = {Hong-Kwang Jeff Kuo and Zoltán Tüske and Samuel Thomas and Brian Kingsbury and George Saon},
  title = {{Improving End-to-End Models for Set Prediction in Spoken Language Understanding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=P54DUpqa7K},
}
Figure