TL;DR
This paper tackles the challenge of data scarcity in spoken semantic parsing (SSP) by leveraging unpaired text data and synthetic data generated by Large Language Models (LLMs). They demonstrate that using unpaired text with methods like Joint Audio-Text (JAT) and Text-to-Speech (TTS) significantly improves SSP performance, especially for new domains. Furthermore, they show that LLM-generated synthetic data can effectively augment existing datasets and enable SSP for unseen domains.
What Problem Does This Paper Address?
Training robust Spoken Semantic Parsing (SSP) models, which convert spoken language into machine-understandable structures, typically requires large datasets of matched speech, transcript, and semantic parse triplets. Acquiring such data is prohibitively expensive and time-consuming, limiting the development of models for existing domains and hindering their extension to new ones. This paper addresses this data bottleneck by exploring how to effectively utilize readily available unpaired text data and how to generate synthetic text data using LLMs to improve SSP performance.
Methods
The paper explores two primary avenues for leveraging unpaired text:
- Using existing unpaired text:
- Joint Audio-Text (JAT) Training: This method uses average speech embeddings from paired data to represent unpaired text. It's computationally inexpensive and suitable for on-device applications.
- Text-to-Speech (TTS) Synthesis: This approach generates speech representations from unpaired text by synthesizing speech using TTS models. While potentially more computationally intensive, it can capture richer speech information.
- Generating synthetic unpaired text with LLMs:
- For Existing Domains (ED): LLMs (specifically Llama 2.0) are prompted to generate transcripts. These transcripts are then pseudo-labeled using a pre-trained semantic parser. Two prompting strategies are used:
- Intent-Word-based Prompting (IWP): Prompts include specific intents and words that frequently co-occur with those intents.
- Exemplar-based Prompting (EP): Prompts provide a few example utterances to guide the LLM in generating similar, diverse transcripts.
- For New Domains (ND): Since pre-trained parsers are unavailable, LLMs are prompted to generate "seqlogical forms" (transcript with intent and slot tags). The transcript is then inferred by removing the tags. Exemplar-based prompting is used here, leveraging a few real examples for each intent-slot combination.
The generated synthetic text data is then used with JAT or TTS to train the SSP models.
Datasets
- STOP Dataset: This is the primary dataset used for training and evaluation. It's a public dataset containing 100 hours of real speech for spoken semantic parsing, covering 8 domains, 28 intents, and 82 slot types. The paper uses it for both existing and new domain experiments.
Key Insights and Ideas
- Unpaired text is a powerful data augmentation tool: Even without corresponding speech, unpaired text can significantly boost SSP performance, especially when dealing with new domains where labeled data is scarce.
- JAT vs. TTS trade-offs: For existing domains, JAT offers a good balance of performance improvement and computational efficiency. For new domains, TTS shows a stronger advantage, likely due to its ability to capture more nuanced speech characteristics that help overcome ASR errors.
- LLMs can effectively generate synthetic data for SSP: Prompting LLMs with carefully designed strategies (IWP and EP) can yield useful synthetic transcripts and seqlogical forms, enabling SSP model training even when no existing textual corpora are available.
- Exemplar-based prompting is versatile: EP proves effective for both generating transcripts in existing domains and seqlogical forms in new domains, demonstrating its flexibility.
- Synthetic data bridges the gap for new domains: LLM-generated data, when combined with TTS, can significantly improve performance on unseen domains, bringing it closer to the performance achieved with real data.
Key Results
- Unpaired text from existing corpora: Improves Exact Match (EM) by 2% and 30% absolute for existing and new domains, respectively, when used with JAT or TTS.
- LLM-generated data for existing domains: Augmenting the STOP baseline with LLM-generated data (using JAT) improved EM by 1.4% absolute. Combining IWP and EP strategies yielded the best results, achieving 68.75% EM (No Err).
- LLM-generated data for new domains: Using LLM-generated data with TTS for a new domain (weather) improved EM by 2.6% absolute over a baseline with limited real data. The performance with synthetic data was close to, but still lagged behind, a topline using real data.
Limitations and Future Work
- LLM-generated data quality: While effective, the LLM-generated data for new domains doesn't perfectly match real speech, leading to a performance gap compared to models trained on real data. The paper notes that the generated data doesn't have matching real speech for the ND setup.
- Prompting strategy optimization: The paper explores IWP and EP, but further research into more sophisticated prompting techniques could yield even better synthetic data.
- Computational cost of TTS: While JAT is efficient, TTS can be computationally expensive, especially for large-scale data generation.
- Future Work: The authors suggest exploring more advanced prompting strategies for LLMs and investigating methods to further bridge the performance gap between synthetic and real data, particularly for new domains. They also highlight the potential for using LLM-generated data to improve ASR performance itself.
Summary updated