Analysis of Mismatched Transcriptions Generated by Humans and Machines for Under-Resourced Languages

Van Hai Do, Nancy F. Chen, Boon Pang Lim, Mark Hasegawa-Johnson

Introduction

Of the thousands of languages spoken globally, only a tiny fraction possess the labeled audio data required to train modern Automatic Speech Recognition (ASR) systems. For these "under-resourced" languages, hiring native transcribers is often prohibitively expensive or logistically impossible. This paper investigates mismatched crowdsourcing: a technique where individuals who do not speak a target language provide phonetic transcriptions using their own native phonetics (e.g., an English speaker transcribing Vietnamese as a sequence of English-like nonsense words). The study specifically explores whether machines—cross-lingual ASR systems—can perform this task as effectively as humans and how these disparate data sources can be combined to improve recognition accuracy.

The Mismatched Channel Framework

The researchers frame the transcription process as a communication problem where the non-native listener acts as a "noisy channel."

The mismatched channel model illustrating how non-native transcriptions (Y) are decoded back into target language phonemes (X-tilde).

As illustrated in the figure above, a native speaker's utterance ($S$) is transcribed by a non-native listener into a sequence ($Y$). A decoder, typically implemented using Weighted Finite State Transducers (WFST), then maps these "nonsense" strings back to the most likely phoneme sequence in the target language ($ ilde{X}$). This allows researchers to utilize data from non-experts to train models for languages like Vietnamese.

Methods and Source Selection

The study compares human transcribers (English and Mandarin speakers) against ASR systems trained on resource-rich languages (Czech, Hungarian, Russian, English, and Mandarin). Two primary innovations are introduced:

  1. Normalized Entropy ($H_{norm}$): To predict which foreign ASR system will best transcribe a new language without needing ground-truth labels, the authors measure the "sharpness" of the ASR's internal probability distributions (posteriorgrams). This metric showed a 0.976 correlation with actual Phoneme Error Rate (PER).
  2. Human-Machine Combination: The researchers developed a pipeline to merge human and machine outputs. By treating the machine as another "mismatched" worker, they can create a more robust training set.
System architecture for merging human and machine-generated mismatched transcriptions through a converter and merger process.

Key Insights and Results

  • Linguistic Proximity: Mandarin transcribers and ASR models significantly outperformed English counterparts when transcribing Vietnamese, likely due to shared syllable structures and linguistic similarities.
  • Complementary Errors: While humans and machines exhibit similar broad confusion patterns, their specific errors are often distinct. Machines frequently capture phonologically meaningful boundaries that human non-native listeners might overlook.
  • Performance Gains:
    • Human Only: Mandarin transcribers achieved a 69.20% PER.
  • Machine Only: The best machine system (Hungarian) achieved a 75.42% PER.
  • Synergy: Combining Mandarin human and machine transcriptions reduced the PER to 67.73%. By converting machine output to a "human-like" format before merging, the researchers achieved an optimal 66.40% PER.

Conclusion and Future Work

The study demonstrates that machine-generated mismatched transcriptions are a viable and complementary alternative to human crowdsourcing for under-resourced languages. Future research will focus on addressing tonal information in languages like Vietnamese, which was simplified in this study, and streamlining the automated integration of these multi-source information streams.

Summary updated

Abstract

When speech data with native transcriptions are scarce in an under-resourced language, automatic speech recognition (ASR) must be trained using other methods. Semi-supervised learning first labels the speech using ASR from other languages, then re-trains the ASR using the generated labels. Mismatched crowdsourcing asks crowd-workers unfamiliar with the language to transcribe it. In this paper, self-training and mismatched crowdsourcing are compared under exactly matched conditions. Specifically, speech data of the target language are decoded by the source language ASR systems into source language phone/word sequences. We find that (1) human mismatched crowdsourcing and cross-lingual ASR have similar error patterns, but different specific errors. (2) These two sources of information can be usefully combined in order to train a better target-language ASR. (3) The differences between the error patterns of non-native human listeners and non-native ASR are small, but when differences are observed, they provide information about the relationship between the phoneme systems of the annotator/source language (Mandarin) and the target language (Vietnamese).
BibTeX
@inproceedings{do2016analysis,
  author = {Van Hai Do and Nancy F. Chen and Boon Pang Lim and Mark Hasegawa-Johnson},
  title = {{Analysis of Mismatched Transcriptions Generated by Humans and Machines for Under-Resourced Languages}},
  booktitle = {Interspeech},
  year = {2016},
  doi = {10.21437/interspeech.2016-736},
}
Figure