Acquiring Speech Transcriptions Using Mismatched Crowdsourcing

Preethi Jyothi, Mark Hasegawa-Johnson

Introduction

Building modern speech recognition systems requires massive amounts of transcribed audio. Traditionally, this necessitates hiring native speakers—a major bottleneck for "low-resource" languages where such speakers might not be easily accessible via global crowdsourcing platforms like Amazon Mechanical Turk. This paper explores a provocative alternative: Can humans who don't know a language still help transcribe it? By treating non-native crowd workers as a "noisy channel" and applying information-theoretic repetition codes, the researchers successfully recovered Hindi words with over 85% accuracy using only English-speaking transcribers.

The Noisy Channel Model

The researchers frame the transcription process using a Noisy Channel Model, a classic concept from communication theory. In this framework, the transcription process is decomposed into several stages:

  1. The Channel: A native speaker produces a word ($X$). This signal passes through a "channel"—the perception and transcription of an English-speaking crowd worker—who outputs an English letter sequence ($Y$).
  2. Repetition Coding: To overcome the high error rate of a single non-native listener, the authors use a "repetition code," asking multiple independent workers to transcribe the same audio clip.
The noisy channel model with repetition coding, showing how a single input is processed by multiple independent channel invocations.
  1. Probabilistic Decoding: The system uses Finite State Transducers (FSTs) to model the mapping between the target language phonemes and the transcriptions. These models learn how non-native speakers systematically perceive foreign sounds. To recover the original word, the decoder searches for the candidate that most likely produced the observed set of mismatched transcriptions.

Experimental Setup

The study utilized approximately one hour of Hindi speech from SBS radio podcasts. A vocabulary of 2,444 words was derived from this audio and the EMILLE corpus. The "crowd" consisted of 134 Amazon Mechanical Turk workers, primarily native English speakers with no knowledge of Hindi. For evaluation, 200 isolated Hindi words were each transcribed by 15 different workers.

Key Insights

  • Systematic Noise: Non-native errors are not random; they are governed by the workers' native phonology. For instance, English speakers may consistently map specific Hindi aspirated stops to the closest English equivalent. Because this "noise" is predictable, it can be mathematically modeled and reversed.
  • Preserving Information: Accuracy improved significantly when workers were allowed to type "nonsense" syllables rather than being forced to use real English words, as the latter often obscured raw phonetic information.

Results and Performance

The system achieved high accuracy by leveraging the "wisdom of the mismatched crowd." With a 15-fold repetition code, the correct Hindi word was recovered with 85.5% accuracy. Furthermore, the correct word appeared in the "top 4" suggestions 94% of the time.

Error rates for N-best suggestions as a function of the number of crowd workers, illustrating diminishing returns.

As shown in the performance analysis, the most significant gains in accuracy occur within the first 5–7 transcribers. Beyond this point, adding more workers provides smaller incremental improvements, suggesting an optimal balance between cost and precision for practical applications.

Limitations and Future Directions

While successful for isolated words, transcribing continuous speech remains a challenge due to blurred word boundaries and an exponentially larger search space. Future work could involve integrating N-gram language models to boost accuracy or weighting crowd workers based on their individual reliability. Nonetheless, this framework demonstrates that mismatched crowdsourcing is a viable path for bootstrapping speech resources in low-resource environments.

Summary updated

Abstract

Transcribed speech is a critical resource for building statistical speech recognition systems. Recent work has looked towards soliciting transcriptions for large speech corpora from native speakers of the language using crowdsourcing techniques. However, native speakers of the target language may not be readily available for crowdsourcing. We examine the following question: can humans unfamiliar with the target language help transcribe? We follow an information-theoretic approach to this problem: (1) We learn the characteristics of a noisy channel that models the transcribers' systematic perception biases. (2) We use an error-correcting code, specifically a repetition code, to encode the inputs to this channel, in conjunction with a maximum-likelihood decoding rule. To demonstrate the feasibility of this approach, we transcribe isolated Hindi words with the help of Mechanical Turk workers unfamiliar with Hindi. We successfully recover Hindi words with an accuracy of over 85% (and 94% in a 4-best list) using a 15-fold repetition code. We also estimate the conditional entropy of the input to this channel (Hindi words) given the channel output (transcripts from crowdsourced workers) to be less than 2 bits; this serves as a theoretical estimate of the average number of bits of auxiliary information required for errorless recovery.
BibTeX
@inproceedings{jyothi2015acquiring,
  author = {Preethi Jyothi and Mark Hasegawa-Johnson},
  title = {{Acquiring Speech Transcriptions Using Mismatched Crowdsourcing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9343},
}
Figure