Deriving conversation-based features from unlabeled speech for discriminative language modeling

Damianos G. Karakos, Brian Roark, Izhak Shafran, Kenji Sagae, Maider Lehr, Emily Tucker Prud'hommeaux, Puyang Xu, Nathan Glenn, Sanjeev Khudanpur, Murat Saraclar, Daniel M. Bikel, Mark Dredze, Chris Callison-Burch, Yuan Cao, Keith B. Hall, Eva Hasler, Philipp Koehn, Adam Lopez, Matt Post, Darcey RileyInterspeech 2012 · OpenReview

Abstract

This paper introduces a novel semi-supervised approach to improve Automatic Speech Recognition (ASR) by using unlabeled audio data to guide a discriminative language model (DLM). By extracting high-level features like topic distributions and document similarities from vast amounts of unlabeled speech, the authors significantly reduce Word Error Rates (WER), outperforming standard supervised models.

Introduction

In Automatic Speech Recognition (ASR), a discriminative language model (DLM) acts as a second-pass filter. It takes the top $N$ hypotheses from a speech recognizer and re-ranks them to minimize word errors. Traditionally, these models are trained using a perceptron algorithm on labeled data—audio paired with manual transcripts.

However, this approach faces two primary challenges:

  1. Data Scarcity: Labeled transcripts are expensive and time-consuming to produce.
  2. Generalization: DLMs often rely on $n$-gram features. If specific word sequences in test data were absent from the small labeled training set, the model struggles to generalize.

This research explores whether massive amounts of unlabeled audio data can help the model understand "well-formed" or "likely" sentences within a specific conversational context.

Methods

The researchers utilize a perceptron-based re-ranking framework. While the perceptron is a supervised learner, it is augmented with "semi-supervised" features derived from 1,545 hours of unlabeled conversational speech.

Unlike typical semi-supervised learning where labeled data is used to guess labels for unlabeled data, this approach uses the global structure of unlabeled data to provide "well-formedness" estimates for hypotheses in the labeled training set.

Feature Engineering

  • Tf-idf Similarity: Each conversation side is treated as a "document." The similarity of an ASR hypothesis to the broader unlabeled corpus is calculated using cosine similarity.
  • Topic-Model Features: Topic models (e.g., 50, 100, or 200 topics) are built from the unlabeled data using the Mallet toolkit. The likelihood of a hypothesis is then calculated given the inferred topic of the conversation.
  • Clustering Features: A graph of unlabeled documents is built based on similarity, and "normalized cuts" are used for clustering. Symbolic features then link specific words to these clusters.
  • Quantized Rank Features: Real-valued scores (similarity or likelihood) are converted into "bins" or ranks to allow the perceptron to handle non-linear information effectively.

Experimental Setup

The study utilized a corpus of Conversational Telephone Speech (CTS) totaling 2,000 hours:

  • Labeled Training Data: ~337 hours (Fisher, Switchboard, Call Home) used to train the perceptron.
  • Unlabeled Data: ~1,545 hours used to derive semi-supervised features.
  • Evaluation: The standard Dev04f and Eval04f datasets were used for tuning and blind testing.

Key Results

The addition of semi-supervised features led to statistically significant improvements in ASR accuracy. The transition from a baseline perceptron using only 3-gram features to one incorporating semi-supervised features resulted in a notable reduction in Word Error Rate (WER).

Word Error Rate (WER) comparison across different model configurations on dev and eval sets.

As shown in the results, the model using semi-supervised features achieved a WER of 24.8% on the evaluation set, compared to the 25.1% baseline. While a 0.3% absolute reduction may seem modest, it represents a significant gain in competitive ASR benchmarks, especially as it leverages "free" unlabeled data rather than expensive manual transcriptions.

Discussion and Conclusion

The core innovation of this work is the "reverse" semi-supervised flow, where unlabeled data serves as a source of truth for lexical coherence. By looking at aggregate patterns across 1,500+ hours of speech, the model learns context better than it could from a small labeled set. While $n$-grams provide local context, tf-idf and topic features offer a global view, helping the model reject linguistically possible but contextually inappropriate hypotheses.

Future work may involve exploring more diverse features or integrating these principles into modern neural architectures. Although this work predates the dominance of Transformer models, the principle of using massive unlabeled data for contextual well-formedness remains a cornerstone of modern self-supervised learning.

Summary updated

BibTeX
@inproceedings{karakos2012deriving,
  author = {Damianos G. Karakos and Brian Roark and Izhak Shafran and Kenji Sagae and Maider Lehr and Emily Tucker Prud'hommeaux and Puyang Xu and Nathan Glenn and Sanjeev Khudanpur and Murat Saraclar and Daniel M. Bikel and Mark Dredze and Chris Callison-Burch and Yuan Cao and Keith B. Hall and Eva Hasler and Philipp Koehn and Adam Lopez and Matt Post and Darcey Riley},
  title = {{Deriving conversation-based features from unlabeled speech for discriminative language modeling}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=rWmFKqr-q2},
}
Figure