TL;DR
This paper compares two Automatic Speech Recognition (ASR) approaches, Elpis (Kaldi-based) and Whisper (transformer-based), for transcribing Čakavian, a low-resource endangered language. Whisper, despite requiring more computational resources, significantly outperformed Elpis, demonstrating the power of large pre-trained models for adapting to under-resourced languages.
What Problem Does This Paper Address?
Documenting endangered languages is crucial, but the lack of readily available tools, especially for Automatic Speech Recognition (ASR), poses a significant hurdle. Many low-resource languages, like Čakavian (a variety of Croatian with around 50,000 speakers), lack the extensive datasets and pre-trained models needed to build effective ASR systems from scratch. This scarcity is exacerbated by linguistic variation within these languages, making transcription even more challenging. This paper aims to evaluate two distinct ASR approaches – a custom-built pipeline (Elpis) and a large pre-trained multilingual model (Whisper) – to determine which is more practical and effective for documenting Čakavian.
Methods
The study compares two ASR systems:
- Elpis: A Kaldi-based pipeline designed for building ASR models from scratch. It uses Mel Frequency Cepstral Coefficients (MFCCs) for feature extraction and allows for n-gram model selection (trigrams were found to be best for this dataset). Elpis is computationally less demanding and requires less training data, making it attractive for resource-constrained environments.
- Whisper: A large, transformer-based multilingual ASR model. The researchers used the
openai/whisper-large-v3model, which was pre-trained on a massive dataset including some Croatian data. The key method here is adaptation (fine-tuning) of this pre-trained model using the limited Čakavian data. Whisper uses log-Mel spectrograms for feature extraction, which are richer than MFCCs.
Both models were trained on the same dataset. For Elpis, a 90%-10% train-test split was used. For Whisper adaptation, an 80%-20% split was employed. Evaluation involved calculating the Word Error Rate (WER) on three separate test interviews.
Datasets
- Training/Adaptation Data: 5.7 hours of Čakavian audio data. This included:
- Five interviews with native Čakavian speakers.
- One audiobook of "The Little Prince" translated into Čakavian.
- Test Data: Three additional Čakavian interviews.
The transcriptions were created by native speakers and linguists with expertise in Čakavian. The data was preprocessed by resampling audio to 16 kHz, standardizing transcriptions (removing punctuation, variable spellings), and handling speech overlap.
Key Insights and Ideas
- Pre-trained models are powerful for low-resource languages: Even with limited domain-specific data, adapting a large pre-trained model like Whisper can yield significantly better results than building a model from scratch with a pipeline like Elpis.
- Data quality and segmentation matter: Whisper's performance was sensitive to factors like audio segmentation window size, the quality of audio data, and speaker overlap. Augmenting with white noise improved performance, likely due to the noisy nature of the interview recordings.
- Elpis is user-friendly but less accurate: While Elpis offers a less computationally expensive and more straightforward user experience, its accuracy for Čakavian transcription was considerably lower.
- Čakavian's distinctiveness is a challenge: The linguistic differences between Čakavian and standard Croatian, including phonology, morphology, syntax, and lexicon, pose a significant challenge for ASR systems not specifically trained or adapted for it.
Key Results
- Whisper significantly outperformed Elpis: The best-performing adapted Whisper model (
whisper-ckm-3) achieved a median WER of 35.48% and an average WER of 57.3%. - Elpis's performance was poor: Elpis achieved a median WER of 129% and an average WER of 130.1%. The high WER for Elpis was largely due to numerous insertion errors.
- Error Type Analysis: The best Whisper model showed a 7.5% WER reduction compared to the base Whisper model, with higher correct rates and lower substitution/deletion rates. Elpis exhibited higher deletion and insertion rates.
Limitations and Future Work
- Data limitations: The study acknowledges that the Čakavian data, consisting of sociolinguistic interviews, contained environmental noise and speaker overlap, which are known challenges for ASR. The impact of code-switching, common in such contexts, was not accounted for.
- Comparison to other languages: While Elpis's WER was higher than previously reported for other languages, the authors note that this is likely due to the specific challenges of Čakavian and the nature of the collected data.
- Future Directions:
- Further research is needed to understand how different ASR systems handle less-than-ideal training data.
- A formal comparison of ASR-aided workflows versus manual transcription time is suggested.
- The authors note that ASR output is generally considered useful for transcription only if the WER is below 30%, a benchmark not met in this study, highlighting the need for cleaner data or more advanced models.
- Creating cleaner, higher-quality training data for Elpis could potentially lead to better models.
- Sharing adapted Whisper models online is encouraged to benefit the broader research community.
Summary updated