TL;DR
This paper introduces the Althingi ASR system, the first open-source Icelandic speech recognition system designed for parliamentary transcription. It achieves a competitive 7.91% word error rate (WER) in-lab and a 15.0% WER in real-world use, significantly reducing manual transcription effort and improving accessibility of parliamentary records.
What Problem Does This Paper Address?
The Icelandic parliament, Althingi, has a policy of publishing all speeches, which traditionally involves manual transcription and editing. This process is time-consuming and resource-intensive. The paper addresses the challenge of automating this process by developing an Automatic Speech Recognition (ASR) system specifically for Icelandic parliamentary speeches. The goal is to create an ASR system that produces output accurate and readable enough to streamline the work of human editors, thereby reducing the manual effort required to make parliamentary records publicly available.
Methods
The Althingi ASR system comprises two main components: a speech recognizer and an automatic post-processing pipeline.
- Speech Recognizer: The core acoustic model is based on a recipe from the Kaldi ASR toolkit, utilizing a sequence-trained neural network built on factorized time-delay neural networks (f-TDNNs). This model features eleven f-TDNN layers with specific architectural constraints and dimensions. For language modeling, a pruned trigram language model is used for decoding, with options for re-scoring using both 5-gram and recurrent neural network language models (RNN-LMs). While RNN-LMs offer slightly better results, the system currently uses 5-grams for faster processing.
- Automatic Post-processing: This pipeline aims to enhance the raw ASR output for human readability.
- Denormalization and Formatting: The OpenGrm Thrax Grammar Development tool is used to compile grammars into weighted finite-state transducers. This allows for denormalizing numbers, abbreviating words according to parliamentary conventions, collapsing acronyms, and handling other formatting like timestamps and website URLs. Repetitions are also removed.
- Punctuation Restoration: The Punctuator toolkit, a bidirectional recurrent neural network model with an attention mechanism, is employed to restore punctuation marks (periods, commas, colons, question marks) and capitalize sentence beginnings.
- Paragraph Insertion: An adaptation of the punctuation model is used to predict paragraph splits, improving the readability of long speeches by structuring them into smaller units.
Datasets
- Training Data:
- Speech Corpora: Approximately 6300 hours of Icelandic parliamentary speeches were prepared. The current speech recognizer uses a subset of about 1000 hours of cleaned parliamentary speeches. Speed perturbations (sp) were applied to generate an additional 2000 hours of acoustic data.
- Language Models: Trained on the total parliamentary text corpus, comprising 59 million tokens.
- Punctuation Model: Trained on punctuation-annotated text, with approximately 58 million words.
- Paragraph Model: Trained on parliamentary texts, containing roughly 300,000 paragraphs and 52 million words.
- Development and Test Sets:
- Speech: Based on parliamentary speeches from 2016, approximately 11 hours each (94,000 words).
- Punctuation Model: Development and test sets contain 114,000 and 111,000 words, respectively.
- Paragraph Model: Development and test sets contain around 16,500 paragraphs and 2.9 million words each.
Key Insights and Ideas
- Data Quality Over Quantity: The paper highlights that cleaning the training data significantly impacts ASR performance. A model trained on a smaller, meticulously cleaned subset (1000 hours) outperformed one trained on a much larger, less cleaned dataset (6200 hours), suggesting that data quality is a critical factor for Icelandic ASR.
- Linguistic Challenges: Icelandic's highly inflected nature and productive compound word formation pose unique challenges for ASR. A substantial portion of substitution errors involve morphological variations of the same lemma, and compound words being split incorrectly can lead to multiple errors being counted.
- Post-processing is Crucial: While the raw ASR output has a certain WER, the post-processing steps (denormalization, punctuation, paragraphing) are essential for making the output usable by human editors and improving overall perceived quality.
- Trade-offs in Language Modeling: While RNN-LMs offer better WER, the significantly longer processing time and higher memory usage led the authors to opt for faster 5-gram language models in the production system.
Key Results
- In-lab Performance:
- Word Error Rate (WER) of 7.91% achieved using f-TDNN with bi-directional RNN-LM re-scoring on the in-lab test set.
- WER of 8.52% for the production system using f-TDNN with 5-gram re-scoring.
- In-lab F-score for the punctuation model: 80.6%.
- In-lab F-score for the paragraph model: 61.6%.
- Real-world Performance ("in the wild"):
- WER of 15.0 ± 6.0% when including punctuation and other post-processing, tested on over 625 speeches.
- Transcribers graded 77% of the speech transcripts as "Good."
- User Feedback: Editors reported that transcripts are available sooner, and their overall satisfaction with the ASR system has increased compared to earlier versions. Editing time per word was also reduced for "Good" graded speeches.
Limitations and Future Work
- Limitations:
- The "in the wild" WER of 15.0% is an upper limit, as not all mismatches are necessarily true ASR errors.
- The accuracy of the punctuation model, particularly for commas, could be improved.
- The current paragraph insertion model's accuracy is considered tolerable but not high.
- The morphological richness of Icelandic leads to errors that, while not always affecting meaning, increase WER.
- Future Work:
- Investigate the benefit of using pause-annotated data to improve punctuation and paragraph models.
- Explore the use of linguistic information to correct morphological word forms.
- Consider extending the language model with a cache model of pertinent bills to potentially improve recognition.
- Further research into whether a two-stage training for the Punctuator model offers benefits.
Summary updated