TL;DR
This paper explores post-processing noisy ASR transcripts for conversational Latvian, a low-resource language. They demonstrate that even small, fine-tuned LLMs can achieve near state-of-the-art punctuation and capitalization restoration, rivaling much larger models. Furthermore, a fine-tuned Whisper model leveraging acoustic cues outperforms text-only approaches, highlighting the importance of multi-modal information for robust ASR refinement.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems often produce transcripts with errors in punctuation, capitalization, and even mispronounced words. These noisy transcripts significantly hinder readability and downstream Natural Language Processing (NLP) tasks like machine translation or information extraction. While punctuation and capitalization restoration has been a long-standing research area, existing methods often struggle with low-resource languages due to a lack of sufficient training data. Furthermore, conversational speech, with its inherent disfluencies and non-standard phrasing, presents a unique challenge compared to more formal or read speech. This paper aims to bridge this gap by investigating effective post-processing techniques for noisy, conversational Latvian ASR transcripts, focusing on achieving high accuracy with limited resources.
Methods
The researchers compare two main approaches for post-processing:
- Transformer-based Sequence Labeling Models: These models, including BERT-based architectures like RoBERTa and ELECTRA, have been standard for punctuation restoration.
- Large Language Models (LLMs): This includes both large, general-purpose LLMs (like GPT-4) and smaller, fine-tuned LLMs (around 2B parameters). LLMs approach the task generatively, performing text infilling and editing.
A key aspect of their methodology is fine-tuning these models. They explore fine-tuning on a modest text corpus, specifically the LATE-Media corpus for Latvian. They also investigate the impact of acoustic cues by evaluating a fine-tuned Whisper model, which inherently incorporates audio information. For evaluation, they use F1-scores for punctuation and capitalization restoration, and Word Error Rate (WER) to measure unintended transcript modifications. They also perform a detailed error analysis to identify common pitfalls.
Datasets
The primary dataset used for training and evaluation is the LATE-Media corpus. This corpus contains approximately 70 hours of conversational Latvian speech from broadcast recordings, featuring over 250 speakers and diverse speaking styles. The transcriptions are rich, including annotations for non-standard pronunciation and foreign words.
For baselines and comparison, they also reference:
- salieckomatus.lv: A proprietary online service for Latvian punctuation restoration.
- XLM-ROBERTa: A multilingual transformer model trained on Europarl data.
- Whisper-large-v3 and Whisper-large-v3-lv: State-of-the-art end-to-end ASR models, with the latter being fine-tuned on Latvian data.
Key Insights and Ideas
- Small LLMs Pack a Punch: Surprisingly, very small LLMs (around 2B parameters), when fine-tuned, can achieve performance comparable to much larger LLMs on punctuation and capitalization tasks. This is crucial for low-resource settings where deploying massive models is impractical.
- Acoustic Cues Matter: Fine-tuning the Whisper model, which leverages acoustic information, significantly outperforms text-only models, especially on challenging conversational data. This underscores the value of multi-modal approaches for robust ASR refinement.
- Fine-tuning is Key for Consistency: While generative LLMs show promise, they can introduce unintended transcript modifications (higher WER). Fine-tuning drastically reduces these unwanted changes, making the output more reliable.
- Conversational Speech is Tricky: Error analysis reveals that sentence boundary determination and handling speaker disfluencies are persistent challenges, highlighting the need for better annotation and domain adaptation.
Key Results
- Near State-of-the-Art with Small LLMs: Fine-tuned small LLMs (e.g., Gemma-2B, EuroLLM-1.7B) achieved F1 scores for punctuation and capitalization that rivaled larger models, demonstrating their effectiveness.
- Whisper's Advantage: The fine-tuned Latvian Whisper model (whisper-large-v3-lv) achieved a WER of 12.7, significantly outperforming the base Whisper large-v3 model while maintaining strong punctuation and capitalization scores.
- Reduced Unintended Edits: Fine-tuning models, especially those in the 2B parameter range, significantly reduced unintended text changes, with errors often manifesting as ungrammatical forms rather than semantic substitutions.
- LLMs for Error Correction: LLMs showed potential in partially learning error correction and inverse text normalization, correctly replacing approximately 50% of mispronounced words in their test set.
Limitations and Future Work
The authors acknowledge several limitations:
- Single Dataset Evaluation: The models were evaluated on a single Latvian dataset, limiting generalizability to other domains or languages. Future work should involve evaluations on multiple datasets.
- ASR Error Impact: ASR errors significantly impact formatting performance. While data augmentation can help, care must be taken to avoid unintended meaning changes.
- Over-Correction Risk: In sensitive domains (e.g., legal, medical), generative models might introduce edits that subtly alter meaning. Cautious use is advised when exact fidelity to the original speech is paramount.
Future work could involve:
- Extending evaluations to more diverse datasets.
- Investigating more robust methods for handling ASR noise and synthetic errors.
- Developing more precise prompting strategies for LLMs to minimize unintended edits.
- Leveraging LLMs to expedite the creation of formatted transcriptions for existing datasets, such as the LATE-Conversational speech corpus.
Summary updated