Improving Spoken Semantic Parsing using Synthetic Data from Large Generative Models

Roshan S Sharma, Suyoun Kim, Trang T. Le, Daniel A. Lazar, Akshat Shrivastava, Kwanghoon An, Piyush Kansal, Leda Sarı, Ozlem Kalinli, Mike Seltzer2024 · DOI

TL;DR

This paper tackles the challenge of data scarcity in spoken semantic parsing (SSP) by leveraging unpaired text data and synthetic data generated by Large Language Models (LLMs). They demonstrate that using unpaired text with methods like Joint Audio-Text (JAT) and Text-to-Speech (TTS) significantly improves SSP performance, especially for new domains. Furthermore, they show that LLM-generated synthetic data can effectively augment existing datasets and enable SSP for unseen domains.

What Problem Does This Paper Address?

Training robust Spoken Semantic Parsing (SSP) models, which convert spoken language into machine-understandable structures, typically requires large datasets of matched speech, transcript, and semantic parse triplets. Acquiring such data is prohibitively expensive and time-consuming, limiting the development of models for existing domains and hindering their extension to new ones. This paper addresses this data bottleneck by exploring how to effectively utilize readily available unpaired text data and how to generate synthetic text data using LLMs to improve SSP performance.

Methods

The paper explores two primary avenues for leveraging unpaired text:

  1. Using existing unpaired text:
    • Joint Audio-Text (JAT) Training: This method uses average speech embeddings from paired data to represent unpaired text. It's computationally inexpensive and suitable for on-device applications.
  2. Text-to-Speech (TTS) Synthesis: This approach generates speech representations from unpaired text by synthesizing speech using TTS models. While potentially more computationally intensive, it can capture richer speech information.
  3. Generating synthetic unpaired text with LLMs:
    • For Existing Domains (ED): LLMs (specifically Llama 2.0) are prompted to generate transcripts. These transcripts are then pseudo-labeled using a pre-trained semantic parser. Two prompting strategies are used:
  4. Intent-Word-based Prompting (IWP): Prompts include specific intents and words that frequently co-occur with those intents.
  5. Exemplar-based Prompting (EP): Prompts provide a few example utterances to guide the LLM in generating similar, diverse transcripts.
  6. For New Domains (ND): Since pre-trained parsers are unavailable, LLMs are prompted to generate "seqlogical forms" (transcript with intent and slot tags). The transcript is then inferred by removing the tags. Exemplar-based prompting is used here, leveraging a few real examples for each intent-slot combination.

The generated synthetic text data is then used with JAT or TTS to train the SSP models.

Datasets

  • STOP Dataset: This is the primary dataset used for training and evaluation. It's a public dataset containing 100 hours of real speech for spoken semantic parsing, covering 8 domains, 28 intents, and 82 slot types. The paper uses it for both existing and new domain experiments.

Key Insights and Ideas

  • Unpaired text is a powerful data augmentation tool: Even without corresponding speech, unpaired text can significantly boost SSP performance, especially when dealing with new domains where labeled data is scarce.
  • JAT vs. TTS trade-offs: For existing domains, JAT offers a good balance of performance improvement and computational efficiency. For new domains, TTS shows a stronger advantage, likely due to its ability to capture more nuanced speech characteristics that help overcome ASR errors.
  • LLMs can effectively generate synthetic data for SSP: Prompting LLMs with carefully designed strategies (IWP and EP) can yield useful synthetic transcripts and seqlogical forms, enabling SSP model training even when no existing textual corpora are available.
  • Exemplar-based prompting is versatile: EP proves effective for both generating transcripts in existing domains and seqlogical forms in new domains, demonstrating its flexibility.
  • Synthetic data bridges the gap for new domains: LLM-generated data, when combined with TTS, can significantly improve performance on unseen domains, bringing it closer to the performance achieved with real data.

Key Results

  • Unpaired text from existing corpora: Improves Exact Match (EM) by 2% and 30% absolute for existing and new domains, respectively, when used with JAT or TTS.
  • LLM-generated data for existing domains: Augmenting the STOP baseline with LLM-generated data (using JAT) improved EM by 1.4% absolute. Combining IWP and EP strategies yielded the best results, achieving 68.75% EM (No Err).
  • LLM-generated data for new domains: Using LLM-generated data with TTS for a new domain (weather) improved EM by 2.6% absolute over a baseline with limited real data. The performance with synthetic data was close to, but still lagged behind, a topline using real data.

Limitations and Future Work

  • LLM-generated data quality: While effective, the LLM-generated data for new domains doesn't perfectly match real speech, leading to a performance gap compared to models trained on real data. The paper notes that the generated data doesn't have matching real speech for the ND setup.
  • Prompting strategy optimization: The paper explores IWP and EP, but further research into more sophisticated prompting techniques could yield even better synthetic data.
  • Computational cost of TTS: While JAT is efficient, TTS can be computationally expensive, especially for large-scale data generation.
  • Future Work: The authors suggest exploring more advanced prompting strategies for LLMs and investigating methods to further bridge the performance gap between synthetic and real data, particularly for new domains. They also highlight the potential for using LLM-generated data to improve ASR performance itself.

Summary updated

Abstract

Spoken semantic parsing (SSP) involves generating machine-comprehensible parses from input speech.Training robust models for existing application domains represented in training data or extending to new domains requires corresponding triplets of speech-transcript-semantic parse data, which are expensive to obtain.In this paper, we address this challenge by examining methods that can use transcript-semantic parse data (unpaired text) without the corresponding speech.First, when unpaired text is drawn from existing textual corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways to generate speech representations for unpaired text.Experiments on the STOP dataset show that unpaired text from existing and new domains improves performance by 2% and 30% in absolute Exact Match (EM) respectively.Second, we consider the setting when unpaired text is not available in existing textual corpora.We propose prompting Large Language Models (LLMs) to generate unpaired text for existing and new domains.Experiments show that examples and words that cooccur with intents can be used to generate unpaired text with Llama 2.0.Using the generated text with JAT and TTS for spoken semantic parsing improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains, respectively.
BibTeX
@misc{sharma2024improving,
  author = {Roshan S Sharma and Suyoun Kim and Trang T. Le and Daniel A. Lazar and Akshat Shrivastava and Kwanghoon An and Piyush Kansal and Leda Sarı and Ozlem Kalinli and Mike Seltzer},
  title = {{Improving Spoken Semantic Parsing using Synthetic Data from Large Generative Models}},
  year = {2024},
  doi = {10.21437/syndata4genai.2024-15},
}
Figure