Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus

Yi-Hui Chou, Kalvin Chang, Meng-Ju Wu, Winston Ou, Alice Wen-Hsin Bi, Carol Yang, Bryan Y. Chen, Rong-Wei Pai, Po-Yen Yeh, Jo-Peng Chiang, Iu-Tshian Phoann, Winnie Chang, Chenxuan Cui, Noel Chen, Jiatong Shi

Abstract

Taiwanese Hokkien is declining in use and status due to a language shift towards Mandarin in Taiwan. This is partly why it is a low resource language in NLP and speech research today. To ensure that the state of the art in speech processing does not leave Taiwanese Hokkien behind, we contribute a 1.5-hour dataset of Taiwanese Hokkien to ML-SUPERB's hidden set. Evaluating ML-SUPERB's suite of self-supervised learning (SSL) speech representations on our dataset, we find that model size does not consistently determine performance. In fact, certain smaller models outperform larger ones. Furthermore, linguistic alignment between pretraining data and the target language plays a crucial role.

Introduction

Taiwanese Hokkien, spoken by approximately 18 million people, faces a significant challenge in the digital age. Despite its large speaker base, it remains a "low-resource" language in the context of Artificial Intelligence and Natural Language Processing (NLP). A historical shift toward Mandarin in Taiwan has resulted in a scarcity of high-quality, transcribed speech data, which is essential for training modern Automatic Speech Recognition (ASR) systems.

Beyond data scarcity, the language presents unique linguistic hurdles. It is not mutually intelligible with Mandarin, despite sharing Sinitic roots. It features a complex system of "tone sandhi," where syllable tones change based on their position in a sentence. Additionally, the lack of a standardized writing system—with speakers alternating between Chinese characters (Han-lo) and Romanization (Tâi-lô)—complicates the development of consistent benchmarks.

Methodology and Dataset Creation

To address these gaps, this research introduces a new 1.5-hour dataset curated from 31 Taiwanese television dramas. This "Taiwanese Drama Corpus" provides a diverse range of prosodic features, including emotive speech like crying and laughing, which serves as a rigorous "stress test" for speech models.

Data Processing Pipeline

Creating a high-quality dataset from television media required several technical steps:

  1. Source Separation: The researchers utilized Demucs, a source separation tool, to isolate clean speech from background music and sound effects common in dramas.
  2. MT-Assisted Annotation: To accelerate transcription, a Machine Translation (MT) model was used to generate draft Taiwanese transcriptions from Mandarin subtitles. These drafts were then meticulously corrected by fluent annotators.
  3. Phonetic Tokenization: Instead of relying on raw characters, the team tokenized syllables into "initials" and "finals." This approach, standard in Sinitic phonology, allows models to share information across different words that share phonetic components.

Evaluation Framework

The study evaluated various Self-Supervised Learning (SSL) speech representations. Rather than training models from scratch, the researchers utilized "upstream" models—such as wav2vec 2.0 and HuBERT—that were pretrained on massive datasets. These models were then fine-tuned on the Taiwanese Hokkien dataset. The performance was measured using Syllable Error Rate (SER) and Character Error Rate (CER).

Key Findings

The evaluation yielded several critical insights into how SSL models generalize to low-resource languages:

1. Linguistic Alignment Over Model Size

A primary finding was that the ancestry of the pretraining data is often more important than the raw number of parameters in a model. For instance, the hubert-base-cmn model (pretrained on Mandarin) frequently outperformed larger models pretrained on English or a diverse mix of unrelated languages. This suggests that linguistic proximity between the pretraining and target languages provides a more significant performance boost than sheer scale.

2. The Impact of Data Volume

The transition from 10 minutes to 1 hour of training data resulted in a substantial performance leap. The hubert-large-cmn model, pretrained on 10,000 hours of Mandarin, achieved the best overall results with an SER of 33.7% when trained on the full 1-hour subset.

3. Challenges in Tone and Phonetics

Models struggled significantly with tone sandhi. Because written transcriptions typically use "base tones" while speakers use "sandhi tones," models must learn a complex internal mapping. Furthermore, phonetic confusion was observed between similar sounds (e.g., [t͡ɕi] vs [d͡ʑi]), likely due to both model limitations and dialectal variations among the drama actors.

Conclusion and Future Directions

This work demonstrates that while SSL models provide a powerful baseline for low-resource languages, they are not a "silver bullet." The performance of these models is heavily dependent on the linguistic alignment of their pretraining data.

Future research will focus on expanding the dataset to several hundred hours to provide a more robust benchmark. Additionally, there is a clear need for ASR architectures that explicitly handle tone sandhi, perhaps through dedicated tone prediction classifiers. By addressing these challenges, we can ensure that Taiwanese Hokkien and other low-resource languages are not left behind in the rapid advancement of speech technology.

Summary updated

Abstract

Taiwanese Hokkien is declining in use and status due to a language shift towards Mandarin in Taiwan. This is partly why it is a low resource language in NLP and speech research today. To ensure that the state of the art in speech processing does not leave Taiwanese Hokkien behind, we contribute a 1.5-hour dataset of Taiwanese Hokkien to ML-SUPERB's hidden set. Evaluating ML-SUPERB's suite of self-supervised learning (SSL) speech representations on our dataset, we find that model size does not consistently determine performance. In fact, certain smaller models outperform larger ones. Furthermore, linguistic alignment between pretraining data and the target language plays a crucial role.
BibTeX
@inproceedings{chou2023evaluating,
  author = {Yi-Hui Chou and Kalvin Chang and Meng-Ju Wu and Winston Ou and Alice Wen-Hsin Bi and Carol Yang and Bryan Y. Chen and Rong-Wei Pai and Po-Yen Yeh and Jo-Peng Chiang and Iu-Tshian Phoann and Winnie Chang and Chenxuan Cui and Noel Chen and Jiatong Shi},
  title = {{Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus}},
  booktitle = {ASRU},
  year = {2023},
  url = {https://openreview.net/forum?id=wzDczSCgr4},
}
Figure