Multilingual Models for ASR in Chibchan Languages

Rolando Coto-Solano, Tai-Wan Kim, Alexander Jones, Sharid LoáicigaNAACL 2024 · DOI

TL;DR

This paper explores building Automatic Speech Recognition (ASR) systems for two extremely low-resource Chibchan languages, Bribri and Cabécar. They demonstrate that transfer learning, particularly from Cabécar to Bribri, significantly improves ASR performance, outperforming joint training and monolingual models. The study also highlights the critical role of data quality and the challenges of transcribing tonal features in complex tonal systems.

What Problem Does This Paper Address?

Developing effective Automatic Speech Recognition (ASR) systems for low-resource languages is a significant challenge. These languages often have limited transcribed audio data, making it difficult to train traditional ASR models from scratch. Furthermore, many indigenous languages, like those in the Chibchan family, possess complex linguistic features such as tonal systems, which are notoriously difficult for ASR models to handle accurately. This paper addresses the need for robust ASR technology for such languages, aiming to facilitate linguistic documentation and revitalization efforts by improving transcription accuracy.

Methods

The researchers experimented with fine-tuning four pre-trained ASR algorithms: Wav2Vec2, Whisper, MMS, and WavLM. They first trained monolingual models for Bribri and Cabécar, identifying Wav2Vec2 as the best-performing base model. Subsequently, they investigated two multilingual learning strategies using Wav2Vec2:

  1. Joint Training: A single model trained simultaneously on data from both Bribri and Cabécar.
  2. Transfer Learning: Models where knowledge gained from training on one language is transferred to improve performance on the other. This was explored in both directions (Bribri-to-Cabécar and Cabécar-to-Bribri).

Additionally, they applied a unigram spell-checking post-processing step to refine the ASR output. A crucial part of their analysis involved an error analysis focused on specific linguistic features, particularly tones, by transforming transcriptions to isolate these elements.

Datasets

The study utilized two Chibchan languages from Costa Rica:

  • Bribri: Approximately 143 minutes of transcribed audio from 28 speakers, sourced from an online oral corpus and a published book.
  • Cabécar: Approximately 54 minutes of transcribed audio from 13 speakers, sourced from a Cabécar dictionary.

The data for both languages was characterized by limited availability and varying degrees of dialectal representation (Bribri had three dialects, Cabécar had one).

Key Insights and Ideas

  • Transfer Learning Dominance: Transfer learning, especially from Cabécar to Bribri, proved to be the most effective strategy, significantly outperforming joint training and monolingual models. This suggests that even limited, high-quality data from a related language can provide substantial benefits.
  • Data Quality is Paramount: The superior performance of transfer learning from Cabécar to Bribri, despite Bribri having more data, highlights the critical impact of data "cleanliness" and consistency. The more consistent Cabécar dataset likely provided a more reliable signal for transfer.
  • Tones are Tricky: The research confirmed that transcribing tones is a significant challenge for ASR models, particularly in languages with complex tonal systems like Bribri. The models made more errors on tonal features compared to consonants or vowels.
  • Spell Checking Nuances: Post-processing spell-checking improved Word Error Rate (WER) for Cabécar but sometimes worsened Character Error Rate (CER) for both languages. This suggests that while spell checkers can fix word-level errors, they might not always improve phonetic accuracy.

Key Results

  • Wav2Vec2 as the Best Base: Wav2Vec2 emerged as the best-performing ASR algorithm for monolingual training on both Bribri and Cabécar.
  • Transfer Learning Gains: The Cabécar-to-Bribri transfer model achieved the lowest error rates for Bribri, with improvements of approximately 2% in CER and 6% in WER compared to the monolingual model.
  • Joint Training Performance: Joint models did not surpass monolingual models, though they performed comparably for Bribri. For Cabécar, joint models showed worse performance than monolingual ones.
  • Tonal Error Analysis: For Bribri, tones were the most difficult feature to transcribe, showing significantly higher error rates (WER=57, CER=30) compared to consonants, vowels, or nasalization. Cabécar's simpler tonal system resulted in lower tonal error rates.
  • Spell Checking Impact: Spell checking reduced WER for Cabécar, but the corrected transcriptions sometimes became less comprehensible due to the unigram nature of the checker.

Limitations and Future Work

  • Limited Speaker Diversity: The models were trained on a relatively small number of speakers, and Cabécar data represented only one dialect. Future work should include a wider diversity of speakers and dialects to ensure broader representativeness.
  • Computational Resources: Training these models, especially the exhaustive experiments, required significant GPU hours, which could be prohibitive for offline implementation in low-resource communities.
  • Data Availability: While Bribri data is open, Cabécar data is only partially available, limiting immediate replication.
  • Future Directions: The authors suggest future work could involve:
  • Incorporating language tags into the models to improve language differentiation.
  • Curating Bribri data to better match the domain of the Cabécar data for improved transfer.
  • Expanding the dataset to include other Chibchan languages (e.g., Malecu, Ngäbere) for zero-shot transcription and building a larger pan-Chibchan multilingual model.
  • Further exploring data processing techniques and the impact of acoustic model saturation.

Summary updated

BibTeX
@inproceedings{cotosolano2024multilingual,
  author = {Rolando Coto-Solano and Tai-Wan Kim and Alexander Jones and Sharid Loáiciga},
  title = {{Multilingual Models for ASR in Chibchan Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.471},
}
Figure