Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems

Samuel Thomas, Hong-Kwang Jeff Kuo, Brian Kingsbury, George Saon

Abstract

The lack of speech data annotated with labels required for spoken language understanding (SLU) is often a major hurdle in building end-to-end (E2E) systems that can directly process speech inputs. In contrast, large amounts of text data with suitable labels are usually available. In this paper, we propose a novel text representation and training methodology that allows E2E SLU systems to be effectively constructed using these text resources. With very limited amounts of additional speech, we show that these models can be further improved to perform at levels close to similar systems built on the full speech datasets. The efficacy of our proposed approach is demonstrated on both intent and entity tasks using three different SLU datasets. With text-only training, the proposed system achieves up to 90% of the performance possible with full speech training. With just an additional 10% of speech data, these models significantly improve further to 97% of full performance.

Introduction

Building robust "end-to-end" (E2E) Spoken Language Understanding (SLU) systems—which identify user intent and extract semantic entities directly from audio—typically requires massive amounts of human-annotated speech data. While raw text data is abundant and easily labeled, paired speech-to-intent data is a significant bottleneck in the development cycle.

Previous research has attempted to bridge this gap using Text-to-Speech (TTS) synthesis to generate synthetic training data or by employing complex multi-modal architectures. However, these methods often introduce high computational overhead or training instability. This paper introduces a more streamlined approach: a single model architecture (RNN-Transducer) that treats text and speech as compatible inputs through a novel representation called a "textogram."

Methodology

The Textogram Representation

The core innovation of this work is the textogram. Standard speech models process log-mel features, which represent sound frequencies over time. To make text compatible with these models, the researchers convert words into a frame-level format. Each character (grapheme) is represented as a one-hot encoding and stretched over a fixed number of time frames, effectively mimicking the temporal nature of speech features.

Training Pipeline

The system utilizes a three-stage training process designed to maximize the utility of available text:

  1. Pre-training: An RNN-Transducer (RNN-T) Automatic Speech Recognition (ASR) model is trained on both real speech and textograms. This forces the encoder to produce similar embeddings regardless of whether the input is acoustic or textual.
  2. Robustness Enhancements: To prevent the model from over-fitting to the "perfect" nature of text, the authors use "Label Masking" (randomly dropping characters) and "Label Confusions" (simulating acoustic similarities, such as swapping 'p' and 'b') to simulate the noise found in real speech.
  3. Adaptation: The pre-trained model is fine-tuned for specific SLU tasks. When only text is available, the encoder is frozen, and only the prediction and joint networks are updated to map the shared embeddings to semantic labels.

Experimental Setup and Datasets

The methodology was evaluated across three distinct SLU benchmarks to test its versatility in intent recognition and entity extraction:

  • HarperValleyBank: A public dataset featuring simulated consumer banking dialogues (1,446 conversations).
  • Call Center Data: An internal IBM dataset consisting of real-world customer utterances categorized into 29 intent classes.
  • ATIS (Airline Travel Information System): A classic benchmark used for both intent recognition and "slot filling" (e.g., identifying "Boston" as a destination).

Key Results

The experimental results demonstrate that textograms can effectively substitute for large portions of speech data:

  • Zero-Speech Performance: On the HarperValleyBank dataset, a model trained on zero speech achieved an F1 score of 45.05. On the Call Center dataset, the text-only model reached 76.97% accuracy, roughly 86% of the performance of a full-speech model.
  • The 10% Threshold: The most significant finding was the impact of adding a small amount of speech data. By adding just 10% of the available speech (e.g., 1.7 hours for HarperValleyBank), performance jumped to 53.84—nearly matching the 54.95 score of the 100% speech model. In the Call Center task, 10% speech data brought the model to 88.34% accuracy, achieving 99% of the full-speech performance.
  • Entity Recognition: On the ATIS dataset, the text-only system achieved an F1 score of 85.95. Interestingly, adding 10% speech data allowed the model to outperform the version trained on 100% speech, suggesting that textograms act as a powerful form of data augmentation.

Discussion

This research highlights two major insights for SLU development. First, modal agnosticism is achievable; by forcing text into a frame-level format, the RNN-T encoder becomes a universal processor, eliminating the need for separate text-processing branches. Second, there are diminishing returns on speech data. Text provides the foundational linguistic knowledge, while a small fraction of speech provides the necessary "acoustic grounding."

Unlike TTS-based methods, this approach does not require generating audio files, significantly reducing computation and storage requirements during training.

Conclusion and Future Work

The proposed textogram method allows for the creation of high-performing SLU systems with minimal speech data, achieving up to 97% of full-scale performance with only 10% of the usual data requirements. While highly effective, the authors noted that performance on datasets like ATIS remains sensitive to speaker coverage. Future work will explore variable label durations to better mimic natural speech rhythms and extend the approach to more complex, multi-turn dialogue systems.

Summary updated

Abstract

The lack of speech data annotated with labels required for spoken language understanding (SLU) is often a major hurdle in building end-to-end (E2E) systems that can directly process speech inputs. In contrast, large amounts of text data with suitable labels are usually available. In this paper, we propose a novel text representation and training methodology that allows E2E SLU systems to be effectively constructed using these text resources. With very limited amounts of additional speech, we show that these models can be further improved to perform at levels close to similar systems built on the full speech datasets. The efficacy of our proposed approach is demonstrated on both intent and entity tasks using three different SLU datasets. With text-only training, the proposed system achieves up to 90% of the performance possible with full speech training. With just an additional 10% of speech data, these models significantly improve further to 97% of full performance.
BibTeX
@inproceedings{thomas2022towards,
  author = {Samuel Thomas and Hong-Kwang Jeff Kuo and Brian Kingsbury and George Saon},
  title = {{Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=INj9uOlbhTo},
}
Figure