TL;DR
This paper tackles the data scarcity problem in training end-to-end Spoken Language Understanding (SLU) models. They propose using speech synthesis (TTS) to generate large, synthetic datasets, demonstrating that these synthetic datasets can be used effectively for training, either alone or to augment limited real-world data, significantly improving SLU model performance.
What Problem Does This Paper Address?
Traditional spoken language understanding (SLU) systems often rely on a pipeline: first, an Automatic Speech Recognition (ASR) system transcribes speech to text, and then a Natural Language Understanding (NLU) module extracts meaning from the text. However, end-to-end SLU models aim to directly map raw audio to semantic meaning, bypassing the intermediate text transcription. This offers advantages like reduced computational complexity and the ability to leverage nuances in speech like prosody.
The major hurdle for end-to-end SLU is its insatiable appetite for in-domain, semantically labeled audio data. Unlike conventional ASR-NLU pipelines that can leverage vast amounts of text data, end-to-end SLU models cannot learn from text alone. This means that for every new domain or application, new audio data must be meticulously recorded and annotated, which is a time-consuming and expensive process. This paper addresses this critical data bottleneck.
Methods
The core of the proposed method is straightforward: leverage existing text data and a Text-to-Speech (TTS) system to create synthetic audio for training end-to-end SLU models. The process involves:
- Text Dataset: A dataset of text utterances paired with their corresponding semantic labels (e.g., intent and slots).
- Speech Synthesis: A TTS system is used to synthesize audio for each text utterance.
- Synthetic Training Data: The synthesized audio, along with its original semantic label, forms the synthetic training dataset.
The authors emphasize three key criteria for selecting a TTS system:
- Multi-speaker: Having multiple synthetic speakers is crucial for achieving high accuracy, mirroring findings in real-world SLU.
- "Everyday" voices: Using casual, non-actor-like voices is preferred to better match the natural speech patterns of users interacting with voice interfaces.
- Open-source: This ensures reproducibility and accessibility for researchers.
They used Facebook's VoiceLoop, an open-source TTS system, which provided 22 synthetic speakers. While acknowledging imperfections in the synthesized speech (vocoder artifacts, punctuation issues, pronunciation errors), they found it effective for training. Their approach can be used as a sole source of training data or as a form of data augmentation by combining synthetic data with limited real audio.
Datasets
The paper uses two open-source SLU datasets for their experiments:
- Fluent Speech Commands (FSC) [17]: This dataset contains 30,043 audio utterances from 77 speakers, labeled with "action," "object," and "location" slots. It's used for experiments on both purely synthetic training and data augmentation.
- Snips SLU Dataset [18]: Specifically, the "smart lights" near-field subset is used. This dataset is smaller and more challenging, featuring variable numbers of slots per utterance and requiring generalization to unseen sentences. It's used to evaluate the effectiveness of synthetic data augmentation on a more difficult task.
Key Insights and Ideas
- Synthetic Data is Viable: The most significant insight is that end-to-end SLU models can be trained effectively using purely synthetic speech, even with TTS imperfections. This dramatically lowers the barrier to entry for developing SLU systems.
- Data Augmentation Power: Synthetic data acts as a powerful augmentation tool. Even with a small amount of real data, adding synthetic data can significantly boost performance, especially when real speakers are scarce.
- Speaker Diversity Matters: The number of synthetic speakers directly impacts accuracy. The performance plateaus after a certain number of speakers (around 15 in their FSC experiments), suggesting that using all available synthetic speakers is generally beneficial.
- Real Speech Still Superior: While synthetic data is valuable, real speech remains the gold standard. Models trained on real speech achieve higher accuracy than those trained solely on synthetic data.
Key Results
- Purely Synthetic Training: On the Fluent Speech Commands dataset, models trained with synthetic speakers achieved an accuracy of 94.9% ± 0.2%, demonstrating the viability of this approach.
- Data Augmentation: When augmenting a real speech dataset for Fluent Speech Commands, adding synthetic speech improved accuracy. For instance, with only three real speakers, the model performed better than using all 22 synthetic speakers alone. The accuracy increased from 65.5% ± 2.9% (real only) to 71.4% ± 1.4% (real + synthetic) on the Snips SLU dataset.
- Speaker Count Impact: Test accuracy on FSC increased significantly up to about 15 synthetic speakers and then plateaued.
- Snips SLU Performance: For the more challenging Snips dataset, including synthetic speakers led to significantly better best accuracy and best loss compared to using real speakers alone.
Limitations and Future Work
- Performance Gap: The authors acknowledge that there is still a performance gap between models trained solely on synthetic speech and those trained on real speech.
- TTS Imperfections: While their method works despite TTS flaws, future work could explore ways to mitigate these imperfections.
- Future Directions:
- Investigate other forms of TTS.
- Develop methods to automatically remove badly synthesized utterances from training sets.
- Combine synthesized speech with other traditional data augmentation techniques (e.g., additive noise, speed perturbation).
Summary updated