ManWav: The First Manchu ASR Model

Jean Seo, Minha Kang, SungJoo Byun, Sangah LeeACL 2024

TL;DR

This paper introduces ManWav, the first Automatic Speech Recognition (ASR) model for Manchu, a critically endangered language. By fine-tuning the multilingual Wav2Vec2-XLSR-53 model on augmented Manchu audio data, they achieve promising results, significantly improving character and word error rates and paving the way for Manchu language preservation.

What Problem Does This Paper Address?

The field of Automatic Speech Recognition (ASR) has historically focused on high-resource languages like English, leading to a significant disparity in research and technological advancements for low-resource languages. This gap is particularly acute for critically endangered languages, where the lack of ASR models hinders preservation efforts and linguistic research. Manchu, a language with no living native speakers, exemplifies this challenge. Existing Manchu research has focused on areas like NER/POS tagging, but a functional ASR model has been absent, leaving a vast amount of unlabeled audio data unused. This paper aims to bridge this gap by developing the first ASR model for Manchu, enabling better utilization of existing data for research and preservation.

Methods

The core of this work involves fine-tuning a pre-trained multilingual self-supervised learning (SSL) model, Wav2Vec2-XLSR-53, for the Manchu language. This model, pre-trained on 53 languages, provides a strong foundation for cross-lingual transfer learning. The authors employed a two-pronged approach:

  1. Fine-tuning with Original Data: The Wav2Vec2-XLSR-53 model was fine-tuned using the original, limited Manchu audio data.
  2. Fine-tuning with Augmented Data (ManWav): To overcome the scarcity of Manchu speech data, they implemented data augmentation techniques using WavAugment. These techniques included:
  3. Additive noise: Introducing background noise.
  4. Clipping: Shortening audio signals.
  5. Reverberation: Applying echo effects.
  6. Time dropout: Randomly removing segments of audio.

This augmentation process expanded the training data by 100% (resulting in a total of 400% of the original data), creating the ManWav model. Crucially, data augmentation was applied after the separation of training and testing sets to ensure reliable evaluation. The fine-tuning process was conducted using an NVIDIA A100 GPU with a learning rate of 3e-4, batch size of 16, and a dropout rate of 0.1.

Datasets

The primary dataset used in this study is Colloquial Manchu data collected as part of the ASK REAL project by Kim et al. (2008). This data originates from the dialect of Sanjiazi village in Heilongjiang Province, China, and consists of basic conversational expressions and sentences for grammatical analysis. The original dataset comprises approximately 90 minutes of audio. The authors also mention that this data underwent revisions by a Manchu researcher for improved accuracy.

Key Insights and Ideas

  • Data Augmentation is Crucial for Low-Resource ASR: The most significant insight is the dramatic improvement in ASR performance achieved through data augmentation. For a language with such limited data, augmenting the existing audio is not just beneficial but essential for building a functional model.
  • Wav2Vec2-XLSR-53's Cross-Lingual Power: The success of fine-tuning a model pre-trained on 53 languages for Manchu highlights the effectiveness of large-scale multilingual SSL models in adapting to even extremely low-resource languages.
  • Manchu's Unique Phonological Challenges: The linguistic analysis reveals specific phonetic confusions that ManWav encounters, such as issues with the schwa sound (/ə/), nasal sounds in word-final positions, assimilation between stops, and confusion between /w/ and /x/. Understanding these challenges is vital for future improvements.
  • Preservation through Technology: The paper underscores the critical role of ASR technology in preserving endangered languages. By making Manchu audio data more accessible and transcribable, ManWav can aid linguists and researchers in their efforts.

Key Results

The experimental results demonstrate the effectiveness of data augmentation:

  • Character Error Rate (CER): Using augmented data resulted in a 0.02 drop in CER compared to using original data.
  • Word Error Rate (WER): Similarly, WER dropped by 0.13 when using augmented data.

Specifically, the Wav2Vec2-XLSR-53 model fine-tuned with augmented data (ManWav) achieved a CER of 0.11 and a WER of 0.31, compared to 0.13 CER and 0.44 WER for the model trained with original data before augmentation. These results are considered promising, especially given that Wav2Vec2-XLSR-53 was not initially pre-trained on Manchu.

Limitations and Future Work

Limitations:

  • Scarcity of Manchu Audio Data: The primary constraint is the limited amount of Manchu audio data available.
  • Single Speaker Data: The audio data used consists of recordings from only one speaker.
  • Domain Dependency: The current ManWav model is trained on colloquial Manchu and may not perform optimally on Manchu spoken in different domains.

Future Work:

  • Decoder Addition: The authors suggest that adding a decoder to the ASR model, known to boost inference performance, should be studied for further enhancement.
  • Language Model Integration: Exploring the use of a language model to improve inference quality is another promising direction.
  • Broader Domain Coverage: Expanding the model's capabilities to handle Manchu in various domains is a key area for future research.
  • Public Release: The authors plan to release ManWav publicly to support further ASR studies on endangered languages.

Summary updated

BibTeX
@inproceedings{seo2024manwav,
  author = {Jean Seo and Minha Kang and SungJoo Byun and Sangah Lee},
  title = {{ManWav: The First Manchu ASR Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure