Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription

Scott Novotney, Chris Callison-BurchNAACL 2010

TL;DR

This paper demonstrates that using crowdsourced transcriptions from Amazon's Mechanical Turk for Automatic Speech Recognition (ASR) training can achieve performance comparable to professional transcriptions at a fraction of the cost. The authors found that investing in more data, rather than solely improving transcription quality, yields better system performance, and they propose a method for estimating transcriber skill without needing a gold standard.

What Problem Does This Paper Address?

Training high-performing Automatic Speech Recognition (ASR) systems, especially for conversational speech, requires vast amounts of transcribed data. This transcription process is notoriously expensive and time-consuming, creating a significant barrier to entry for new languages or specialized domains. While previous work showed that crowdsourcing platforms like Amazon's Mechanical Turk could produce annotations of professional quality for other NLP tasks, this paper specifically investigates its effectiveness for ASR, focusing on reducing transcription costs without sacrificing system performance.

Methods

The core of this research involves training ASR models using transcriptions generated by non-expert workers on Amazon's Mechanical Turk. The authors employed a standard multi-pass LVCSR system (Byblos) with state-clustered Gaussian tied-mixture acoustic models and smoothed language models. They experimented with different transcription strategies:

  • Cost Reduction: They systematically lowered the payment per transcription task on Mechanical Turk to find the sweet spot between cost and quality.
  • Data vs. Quality: They compared the impact of redundant transcriptions (multiple non-experts transcribing the same audio) versus simply collecting more unique audio with single transcriptions.
  • Quality Control: They explored methods for estimating transcriber skill and identifying "good" versus "bad" workers without relying on a pre-existing gold standard transcription. This involved comparing non-professional transcriptions against professional ones and analyzing the correlation.

Datasets

  • English Switchboard Corpus: A 20-hour subset was used for most experiments. This corpus consists of conversations between strangers on assigned topics.
  • NIST HUB5 evaluation (Dev04 test set): A 3-hour set of carefully transcribed data was used for evaluating ASR system performance.
  • LDC (Linguistic Data Consortium) transcriptions: Used as a high-quality "gold standard" for comparison.
  • Fisher quick transcription guidelines: Provided by a professional transcription company, also used as a gold standard.
  • Korean, Hindi, and Tamil CTS data: Collected from the Call-friend corpora for experiments in low-resource languages.

Key Insights and Ideas

  • Cost-Effective Transcription: Mechanical Turk can transcribe conversational speech at approximately 1/30th the cost of professional transcription, making ASR development more accessible.
  • More Data Trumps Higher Quality: For a fixed budget, collecting more data with slightly higher transcription disagreement yields better ASR performance than spending more on higher-quality, but less data, transcriptions.
  • Disagreement Doesn't Always Hurt Performance: While non-professional transcriptions have higher disagreement rates with professional ones, this doesn't translate to a proportional drop in ASR system performance.
  • Estimating Turker Skill is Possible: A practical method for estimating a Turker's skill (disagreement with professional transcriptions) can be developed using only a small number of transcriptions from other non-experts, even when the non-expert reference itself has a known error rate.
  • Quality Control Without Gold Standard: The proposed method for estimating Turker skill allows for quality control and worker selection even in new domains where a gold standard is unavailable.

Key Results

  • ASR systems trained on Mechanical Turk transcriptions (with 23% disagreement) were only 2.5% worse in absolute Word Error Rate (WER) compared to systems trained on high-quality LDC transcriptions.
  • Reducing transcription cost on Mechanical Turk to as low as $5 per hour of transcription still resulted in only an 18% disagreement rate with professionals.
  • Investing in 60 hours of singly transcribed data was more beneficial than using 20 hours of redundantly transcribed data (even with an "utterance oracle" selection).
  • For Korean, a PER of 52.1% was achieved with Mechanical Turk transcripts, only a slight degradation from the 51.3% PER achieved with LDC data.

Limitations and Future Work

  • Low-Resource Languages: While experiments in Korean, Hindi, and Tamil show promise, the performance in these languages was significantly worse than in English, suggesting more work is needed to bootstrap and improve these systems. The authors note that language model improvement through external text sources is crucial for low-resource languages.
  • Quality Control Thresholding: While a method for estimating Turker skill is proposed, the paper acknowledges that finding "good" Turkers is challenging due to a high ratio of false positives to false negatives when rejecting workers based on disagreement. Future work could focus on refining this selection process.
  • Automatic Error Detection: The paper briefly mentions that errant transcripts can be detected by force aligning with audio, but extending this to thousands of hours requires robust, automated methods.
  • Domain Specificity: The research primarily focuses on conversational telephone speech. Its applicability to other types of speech (e.g., broadcast news, lectures) might require further investigation.

Summary updated

BibTeX
@inproceedings{novotney2010cheap,
  author = {Scott Novotney and Chris Callison-Burch},
  title = {{Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure