Getting More Mileage from Web Text Sources for Conversational Speech Language Modeling using Class-Dependent Mixtures

Ivan Bulyko, Mari Ostendorf, Andreas StolckeNAACL 2003 · OpenReview

Abstract

Sources of training data suitable for language modeling of conversational speech are limited. In this paper, we show how training data can be supplemented with text from the web filtered to match the style and/or topic of the target recognition task, but also that it is possible to get bigger performance gains from the data by using class-dependent interpolation of N-grams.

Introduction

Building accurate Automatic Speech Recognition (ASR) systems for conversational speech is notoriously difficult due to a significant data bottleneck. While massive amounts of written text (books, news, Wikipedia) are available, they do not reflect the disfluencies, rhythmic patterns, and informal syntax of human conversation. The "gold standard" training data—manual transcripts of real phone calls or meetings—is expensive and time-consuming to produce. Consequently, researchers often supplement this small "in-domain" pool with "out-of-domain" data like Broadcast News, which often fails to match the informal "vibe" of casual chat. This study explores leveraging the World Wide Web as a conversational training resource through targeted scraping and advanced data blending techniques.

Methods

The authors introduce two primary innovations to overcome the domain gap:

  1. Style-Targeted Web Scraping: Instead of random web downloads, the team used search engines to find text that "sounds" like a conversation. They identified frequent N-grams from the Switchboard corpus—phrases like "I never thought I would"—and used them as search queries. This targeted approach identifies "chat-like" material that mimics human speech better than polished news scripts.
  2. Class-Dependent Mixture Models: Traditional interpolation uses a single weight for an entire data source (e.g., 10% Switchboard + 90% Web). This paper proposes class-dependent interpolation, where weights vary based on the word class (e.g., Part-of-Speech tags) of the preceding word. This allows the model to selectively trust different sources for different syntactic contexts—for instance, trusting conversational transcripts more for filler words like "um" while leveraging web data for noun-adjective sequences.

Experimental Setup

The approach was evaluated on two major benchmarks:

  • HUB5 (eval 2001): A standard set for Conversational Telephone Speech (CTS) based on the Switchboard corpus.
  • ICSI Meeting Recorder: A corpus of spontaneous speech from technical meetings.

Training sources included 3 million words of CTS transcripts, 150 million words of Broadcast News (BN), and up to 191 million words of "conversational" web text.

Key Results and Discussion

The inclusion of style-filtered web data consistently outperformed Broadcast News transcripts for conversational tasks. On the HUB5 benchmark, the baseline WER of 38.9% was reduced to 37.2% by combining web data with class-dependent interpolation.

WER results on HUB5 (CTS) task using various data sources and mixing strategies.

Similar gains were observed in the ICSI Meeting task. By adding meeting-specific and topic-related web data, and applying class-dependent mixing, the WER dropped from a 38.2% baseline to 35.9%.

WER results on the ICSI Meeting task, showing improvements from meeting-specific and topic-related web data.

Beyond WER reductions, the model's perplexity dropped significantly (e.g., from 122 to 95 in the meetings task), indicating a much higher predictive accuracy for conversational flow. The authors also found that automatically clustering words into classes works as effectively as manual Part-of-Speech tagging, making this technique applicable to languages without robust linguistic tools.

Conclusion and Future Work

The study demonstrates that the web is a superior proxy for conversational speech compared to formal news sources when filtered for style. Class-dependent interpolation provides a more granular and effective way to combine disparate data sources by tapping into underlying syntactic structures.

One remaining challenge is the "trimming paradox": further filtering the web corpus using perplexity did not improve results, suggesting that initial style-based filtering already captures the necessary diversity. Additionally, while web text is "chatty," it still lacks the natural disfluencies (like "ums" and "ahs") found in actual speech. Future work may involve scaling these style-filtering techniques to the massive datasets used by modern Large Language Models.

Summary updated

Abstract

Sources of training data suitable for language modeling of conversational speech are limited. In this paper, we show how training data can be supplemented with text from the web filtered to match the style and/or topic of the target recognition task, but also that it is possible to get bigger performance gains from the data by using class-dependent interpolation of N-grams.
BibTeX
@inproceedings{bulyko2003getting,
  author = {Ivan Bulyko and Mari Ostendorf and Andreas Stolcke},
  title = {{Getting More Mileage from Web Text Sources for Conversational Speech Language Modeling using Class-Dependent Mixtures}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
  url = {https://openreview.net/forum?id=BkZ2nz-O-B},
}
Figure