TL;DR
This paper introduces a fully automatic method for aligning large audiobook recordings with their corresponding text transcripts. By combining lightly supervised speech recognition with a novel sentence selection strategy, the approach generates high-quality transcriptions and precise sentence timings, crucial for advancing speech synthesis and analysis research.
What Problem Does This Paper Address?
The availability of large quantities of audio data, such as audiobooks, presents a rich resource for speech research. However, effectively utilizing this data for tasks like training expressive speech synthesizers or analyzing nuanced speech patterns requires accurate transcriptions and precise temporal alignments of speech segments to text. Traditional methods often demand significant manual effort to reconcile discrepancies between spoken audio and written text, especially in long, coherent recordings where speakers might deviate from the script (e.g., adding introductory remarks, skipping sections). This paper aims to bridge this gap by providing a fully automatic solution for generating these alignments, overcoming the limitations of existing forced alignment techniques.
Methods
The core of the proposed "LightlySupervised" approach lies in a two-stage process:
- Lightly Supervised Recognition:
- Text Normalization: The input text undergoes cleaning, removing headers, footers, and special characters, and is then normalized using a speech synthesis engine (Toshiba's ToSpeak). This engine also generates the lexicon.
- Language Model Creation: A language model is built by combining a general background model (trained on the English Gigaword corpus) with a domain-specific model (trained on the normalized audiobook sentences). This biases the recognition towards the expected content.
- Audio Processing: Audio files are converted to WAV, down-sampled, and converted into Mel-Frequency Cepstral Coefficients (MFCCs). Crucially, large audio files are split into smaller chunks (0.5-30 seconds) for efficient processing.
- Speaker-Independent Recognition: A large vocabulary speech recognition tool (HDecode from HTK) is used with speaker-independent acoustic models.
- Speaker Adaptation: The initial recognition output is used to adapt the speaker-independent models to the specific audiobook narrator using Maximum Likelihood Linear Regression (MLLR). The recognition is then rerun with these adapted models.
- Sentence Selection:
- Alignment and Splitting: The recognized word sequence is aligned with the normalized book text using the longest common subsequence algorithm. This alignment is used to split the recognized output into sentences that correspond to the book's structure.
- Threshold-Based Selection: A word accuracy threshold is applied to select or discard recognized sentences. Sentences with word accuracy above this threshold are retained. This step effectively filters out misrecognized sentences and implicitly handles some text-speech mismatches.
- Timestamp Generation: A final forced alignment is performed on the selected and corrected word sequences to generate precise sentence boundary timestamps.
Datasets
- Training Data: The paper mentions using "over 150h of read speech data (text type was newspaper, i.e. WSJ0 + WSJ1)" for training the initial speaker-independent acoustic models. A background language model was trained on the "English Gigaword corpus."
- Evaluation Data: The primary dataset used for evaluation is the publicly available American audiobook "A Tramp Abroad" by Mark Twain. This audiobook comprises over 15 hours of speech and more than 160,000 words. The "gold standard" (GOLD) transcription was manually created for this audiobook. The paper also mentions applying the method to over 20 other American English audiobooks for evaluation.
Key Insights and Ideas
- Leveraging Text-Speech Overlap: The core idea is that for large coherent corpora like audiobooks, there's a significant overlap between the written text and the spoken word. The method exploits this overlap by using the text to guide and correct the speech recognition process.
- "Light Supervision" is Key: Instead of requiring full manual transcriptions or alignments, the approach uses the existing text as a "light" supervision signal. This significantly reduces the manual effort required.
- Sentence Selection as a Filter: The threshold-based sentence selection is a critical step. It allows researchers to balance the trade-off between the quantity of extracted data and the accuracy of the transcriptions. A higher threshold yields more accurate but fewer sentences, while a lower threshold includes more data but potentially lower accuracy.
- Handling Deviations: The method implicitly handles common deviations like introductory remarks or skipped sections by filtering sentences based on their recognized word accuracy relative to the book text.
- Automaticity is a Major Win: The fully automatic nature of the system is highlighted as a significant advantage over previous methods that required manual intervention.
Key Results
- Reduced Errors: The LightlySupervised approach significantly reduced word insertions (from 3.6% to 1.0%) and sentence insertions (from 6.0% to 0.9%) compared to the raw book text vs. speech comparison.
- High Accuracy: After sentence selection, the word error rate (WER) for extracted sentences could be as low as 0.2% at a 100% word accuracy threshold, which is even lower than the 0.6% upper limit WER achievable when discarding all inserted/deleted text.
- Data Extraction Efficiency: Even at a strict 75% word accuracy threshold, the system extracted 88.9% of sentences from "A Tramp Abroad." At a 100% threshold, it still extracted 65.4% of sentences.
- Accurate Sentence Boundaries: Sentence boundary timestamps were found to be mostly accurate, with over 95% of boundaries being "OK" in a manual evaluation.
Limitations and Future Work
- Limitations:
- The approach does not explicitly handle sentences inserted by the speaker that are not in the book text, though these are implicitly discarded by the selection process.
- The accuracy of the final output is still dependent on the quality of the initial speaker-independent acoustic models and the effectiveness of speaker adaptation.
- The method is primarily designed for English audiobooks and may require adaptation for other languages.
- Future Work:
- Refining acoustic models and improving language model capabilities.
- Developing better mechanisms for handling inserted speech.
- Detecting and separating different speaking styles (e.g., neutral vs. emotional, character voices) within the recordings, which would be valuable for advanced speech synthesis and analysis.
- Exploring confusion networks for more nuanced sentence selection.
Summary updated