Evaluating Self-Supervised Speech Models on a Taiwanese Hokkien Corpus

Yi-Hui Chou, Kalvin Chang, Meng-Ju Wu, Winston Ou, Alice Wen-Hsin Bi, Carol Yang, Bryan Y. Chen, Rong-Wei Pai, Po-Yen Yeh, Jo-Peng Chiang, Iu-Tshian Phoann, Winnie Chang, Chenxuan Cui, Noel Chen, Jiatong Shi

TL;DR

Researchers have developed a new 1.5-hour dataset of Taiwanese Hokkien speech sourced from dramas to bolster low-resource speech processing. Their evaluation of various self-supervised learning (SSL) models reveals that "linguistic alignment"—the similarity between a model's pretraining data and the target language—is often more important for performance than raw model size.

What Problem Does This Paper Address?

Taiwanese Hokkien is spoken by roughly 18 million people, yet it is considered a "low-resource" language in the world of AI and Natural Language Processing (NLP). Because of a historical language shift toward Mandarin in Taiwan, there is a dearth of high-quality, transcribed speech data available for training modern AI models.

Furthermore, Taiwanese Hokkien presents unique linguistic challenges:

  1. Mutual Unintelligibility: While it shares roots with Mandarin, the two are not mutually intelligible.
  2. Complex Phonology: It features a sophisticated system of "tone sandhi," where the tone of a syllable changes depending on its position in a sentence.
  3. Orthographic Inconsistency: There is no single universally used writing system; speakers use various combinations of Chinese characters (Han-lo) and Romanization (Tâi-lô).

This paper aims to provide a standardized benchmark and evaluate how well state-of-the-art speech models—originally trained on massive amounts of English or Mandarin—generalize to this unique language.

Methods

The authors employed a Self-Supervised Learning (SSL) framework. Instead of training a model from scratch, they took "upstream" models (like wav2vec 2.0 or HuBERT) that have already learned the general structure of human speech and "fine-tuned" them on their new Taiwanese dataset.

Key Technical Approaches: * Initials and Finals Tokenization: Rather than using raw characters, they tokenized syllables into "initials" and "finals" (a standard way to analyze Sinitic phonology). This allows the model to share information across different words that share phonetic components. * MT-Assisted Annotation: To speed up the creation of the dataset, the team used a Machine Translation (MT) model to generate draft Taiwanese transcriptions from Mandarin subtitles, which were then manually corrected by fluent annotators. * Source Separation: Since the data came from TV dramas, it often contained background music. The researchers used a source separation tool (Demucs) to strip away music and isolate the clean speech.

Datasets

  • The Taiwanese Drama Corpus (New): A 1.5-hour dataset curated from 31 different Taiwanese soap operas. It includes diverse prosodic features like crying and laughing, making it a "stress test" for speech models.
  • iCorpus: A Mandarin-Taiwanese parallel corpus used to train the machine translation assistant.
  • ML-SUPERB: The new dataset was contributed to the "hidden set" of this multilingual speech benchmark.

Key Insights and Ideas

  • Linguistic Alignment > Model Size: One of the most striking findings was that a smaller model pretrained on Mandarin (hubert-base-cmn) often outperformed much larger models pretrained on English or a diverse mix of unrelated languages. This suggests that for low-resource languages, the ancestry of the pretraining data matters immensely.
  • The "Tone Sandhi" Hurdle: Models frequently struggled with tones. Because the written transcriptions use "base tones" but the speakers use "sandhi tones," the models must learn a complex internal mapping that they weren't originally designed for.
  • Phonetic Confusion: The models often confused phonetically similar sounds (like [t͡ɕi] vs [d͡ʑi]), which the authors attribute to both model limitations and natural dialectal variations among the actors in the dramas.

Key Results

The researchers measured performance using Syllable Error Rate (SER) and Character Error Rate (CER).

  • Top Performer: The hubert-large-cmn model (pretrained on 10k hours of Mandarin) achieved the best results, with an SER of 33.7% when trained on 1 hour of data.
  • The Power of 1 Hour: Increasing the training data from 10 minutes to 1 hour led to a massive performance jump—for example, hubert-large-cmn improved its SER from 45.9% to 33.7%.
  • Baseline Comparison: Standard "fbank" (non-SSL) features performed poorly (78.5% SER), proving that self-supervised representations are essential for making sense of low-resource languages.

Limitations and Future Work

  • Dataset Scale: While 1.5 hours is a valuable start for a benchmark, it is still very small. The authors plan to expand this to a "several hundred hour" dataset.
  • Tone Modeling: Current ASR (Automatic Speech Recognition) architectures don't explicitly handle tone sandhi. The authors suggest that future models might need a separate classifier specifically for tone prediction.
  • Audio Quality: Dramas are "noisy" environments. While source separation helped, overlapping speech and room acoustics remain challenges for the models.

Reference: Chou, Y. H., et al. "Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus." Proceedings of IEEE, 2023.

Summary updated

Abstract

Taiwanese Hokkien is declining in use and status due to a language shift towards Mandarin in Taiwan. This is partly why it is a low resource language in NLP and speech research today. To ensure that the state of the art in speech processing does not leave Taiwanese Hokkien behind, we contribute a 1.5-hour dataset of Taiwanese Hokkien to MLSUPERB’s hidden set. Evaluating ML-SUPERB’s suite of self-supervised learning (SSL) speech representations on our dataset, we find that model size does not consistently determine performance. In fact, certain smaller models outperform larger ones. Furthermore, linguistic alignment between pretraining data and the target language plays a crucial role.
BibTeX
@inproceedings{chou2023evaluating,
  author = {Yi-Hui Chou and Kalvin Chang and Meng-Ju Wu and Winston Ou and Alice Wen-Hsin Bi and Carol Yang and Bryan Y. Chen and Rong-Wei Pai and Po-Yen Yeh and Jo-Peng Chiang and Iu-Tshian Phoann and Winnie Chang and Chenxuan Cui and Noel Chen and Jiatong Shi},
  title = {{Evaluating Self-Supervised Speech Models on a Taiwanese Hokkien Corpus}},
  booktitle = {ASRU},
  year = {2023},
  url = {https://openreview.net/forum?id=wzDczSCgr4},
}

Loading related content...

Figure