TL;DR
This paper explores using a "machine speech chain" (integrating ASR and TTS) to adapt speech models to new domains and speakers. By jointly training ASR and TTS on unpaired target domain text, they achieve significant improvements in both ASR accuracy and TTS naturalness, even with limited target speaker data.
What Problem Does This Paper Address?
Modern speech recognition (ASR) and text-to-speech (TTS) systems have achieved impressive performance, but they often struggle when faced with data that differs from their training set. This "data mismatch" can occur due to variations in speaking style, accent, acoustic environment, or even the domain of the content (e.g., adapting from news to medical transcripts). Traditional adaptation methods often focus on either ASR or TTS individually. This paper tackles the challenge of adapting both ASR and TTS models simultaneously to a new domain, and then further adapting the ASR to new speakers, leveraging the synergistic potential of a "machine speech chain."
Methods
The core of this work is the machine speech chain, which jointly trains an end-to-end ASR model and a neural TTS model. The key idea is that the TTS can generate synthetic speech data for the ASR to train on, and the ASR can provide feedback on the quality of the synthesized speech, helping to improve the TTS.
Specifically, the authors use a TTS→ASR pipeline for domain adaptation. Here's how it works:
- Pre-training: Both ASR and TTS models are initially trained on source domain data (e.g., audiobooks).
- Joint Training for Domain Adaptation:
- The models are then jointly trained using the source domain paired data (speech-text) to calculate standard ASR and TTS losses.
- Crucially, they use unpaired text data from the target domain (e.g., presentation transcripts) to train the TTS→ASR pipeline. The TTS generates speech features for this target text, and the ASR then tries to transcribe these synthetic features. This "TTS→ASR loss" is used to update both models, encouraging them to adapt to the target domain.
- To balance the influence of different loss components, gradient clipping is employed.
- Few-Shot Speaker Adaptation for ASR: After domain adaptation, the TTS model is kept fixed, and the ASR model is further adapted to new speakers using a small number of utterances from those speakers. This is achieved by concatenating speaker embeddings (extracted from a pre-trained speaker model) to the ASR's hidden states.
Datasets
- Source Domain:
- ASR: LibriSpeech (audiobook domain)
- TTS: LibriTTS (audiobook domain)
- Target Domain:
- TED-LIUM (presentation domain) - used for unpaired text data for domain adaptation and as a test set.
- Speaker Adaptation: Test set speakers from TED-LIUM.
The paper highlights the difference between the audiobook domain (formal, strict grammar) and the presentation domain (spoken style, casual grammar), making TED-LIUM a suitable target for demonstrating domain adaptation.
Key Insights and Ideas
- Synergy of ASR and TTS: The machine speech chain isn't just about data augmentation; it's about mutual improvement. TTS can generate diverse acoustic representations for ASR, while ASR can provide a quality signal for TTS, especially for handling unseen text.
- Unpaired Text is Powerful: The ability to adapt to a new domain using only unpaired text from that domain is a significant advantage, as such data is often more readily available than paired speech-text data.
- Domain Adaptation Benefits Both: The joint training approach improves both the ASR's ability to understand target domain content and the TTS's ability to generate more natural and robust speech for that domain.
- Few-Shot Speaker Adaptation: Adapting ASR to new speakers with just a few examples is feasible and effective, further boosting performance beyond domain adaptation.
- Strategy Matters: While joint training of both ASR and TTS for domain adaptation shows strong TTS improvements, updating only the ASR model with the TTS→ASR loss yielded slightly better ASR results in their experiments.
Key Results
- Domain Adaptation for ASR:
- On the TED-LIUM test set, the Word Error Rate (WER) for the ASR baseline was 25.9%.
- After domain adaptation using the machine speech chain, the WER was reduced to 23.3%, a 10% relative reduction.
- Performance on the source domain (LibriSpeech) remained largely unaffected (6.5% vs. 6.6% WER).
- Domain Adaptation for TTS:
- The TTS model showed a significant improvement in robustness for the target domain (TED-LIUM), with a 51.5% relative WER reduction in generated speech.
- Robustness in the source domain also improved by about 16.9%.
- Visualizations of attention alignments showed much better alignment for out-of-domain text after joint training.
- Few-Shot Speaker Adaptation for ASR:
- Using 1 reference utterance for speaker adaptation reduced the WER by an additional 2.6% (from 23.1% to 22.8%).
- Using 5 reference utterances further reduced the WER to 22.5%.
Limitations and Future Work
- Limitations:
- The paper notes that updating only the ASR model during domain adaptation yielded slightly better ASR results than updating both ASR and TTS, suggesting that the TTS updates might not always directly translate to ASR gains in all scenarios.
- The speaker adaptation was applied to ASR only, keeping TTS fixed.
- Future Work:
- The authors plan to combine their proposed method with the ASR→TTS pipeline (using straight-through estimators) to explore unsupervised pre-training of both ASR and TTS.
- Further exploration of few-shot speaker adaptation for TTS is also a potential direction.
Summary updated