A bootstrapping approach for developing language model of new spoken dialogue systems by selecting web texts

Teruhisa Misu, Tatsuya KawaharaInterspeech 2006

TL;DR

This paper introduces a novel "bootstrapping" method to create language models for new spoken dialogue systems by intelligently selecting relevant text from the World Wide Web. By using existing knowledge bases as seed queries and a diverse dialogue corpus to filter for spoken style, they significantly reduce the word error rate in speech recognition for new domains, even without in-domain training data.

What Problem Does This Paper Address?

Developing effective language models (LMs) for spoken dialogue systems is crucial for their performance. Traditionally, this requires large amounts of domain-specific user utterance data, which is expensive and time-consuming to collect. Handcrafting grammars or using Wizard-of-Oz methods are also costly and can be unreliable. While the World Wide Web offers a vast text resource, simply scraping it for data isn't enough, as web text is often in a written style, not matching the spoken utterances of users. This paper tackles the challenge of efficiently building high-quality LMs for new spoken dialogue domains when in-domain utterance data is scarce, by leveraging readily available web content and existing dialogue corpora.

Methods

The core of the proposed method is a bootstrapping approach that iteratively builds a language model. Here's a breakdown:

  1. Seeding with Knowledge Bases (KB): Instead of relying on existing dialogue data, the method starts with a knowledge base (like a document set or a knowledge graph) relevant to the target domain. Keywords are extracted from this KB to form search queries.
  2. Web Text Retrieval: These queries are used to search the World Wide Web, retrieving a large volume of text pages.
  3. Baseline Model Construction: A crucial step involves creating a "baseline" language model. This model is built by combining the initial KB texts with an existing spoken dialogue corpus from a different domain. This corpus, though not domain-specific, provides a model of spoken utterance style.
  4. Sentence Selection: The baseline model is then used to score the retrieved web texts. Sentences that are more similar to the spoken style (indicated by higher model likelihood or lower perplexity) are selected. A threshold ($\theta$) is used to filter these sentences.
  5. Final Language Model Training: The final language model for the target domain is trained by combining the KB texts, the existing dialogue corpus, and the selected web texts.

This process effectively uses the KB to find domain-relevant content and the external dialogue corpus to filter for appropriate spoken style, bootstrapping the LM development.

Datasets

  • Knowledge Bases (KB):
    • Software Support Task: Microsoft's software support documents.
  • Sightseeing Guidance Task: Official tourist information for Kyoto city and Wikipedia documents about Kyoto.
  • Existing Dialogue Corpus (Different Domain): The CIAIR in-car spoken dialogue corpus, which consists of restaurant search dialogues. User-side utterances were extracted for style matching.
  • Evaluation Data:
    • Software Support Task: 499 utterances from 30 users.
  • Sightseeing Guidance Task: 220 utterances from 4 users.

Key Insights and Ideas

  • Leveraging KBs as Seeds: Using domain-specific KBs as initial seeds for web search queries is a practical way to gather relevant content without needing in-domain utterance data.
  • The Power of Style Matching: The most significant insight is the critical role of an external, style-matched dialogue corpus (like the CIAIR corpus) in filtering web texts. Simply using domain relevance isn't enough; the text must also sound like natural speech. This external corpus acts as a proxy for spoken style.
  • Bootstrapping is Effective: The method demonstrates that a robust language model can be built from scratch for a new domain, relying on readily available resources rather than extensive in-domain data collection.
  • Filtering is Essential: Using all retrieved web texts can degrade performance. Intelligent selection based on style is paramount.
  • Comparable to Large Datasets: The proposed method achieves performance comparable to systems trained on millions of domain-specific sentences, highlighting its efficiency.

Key Results

The proposed bootstrapping method, denoted as (KB+CIAIR+(Web by KB & CIAIR)), showed significant improvements in speech recognition performance, measured by Word Error Rate (WER):

  • Software Support Task: Achieved a 5.7% absolute reduction in WER compared to baseline methods.
  • Sightseeing Guidance Task: Achieved a 5.8% absolute reduction in WER.

The paper also showed that:

  • Using the external dialogue corpus for sentence selection led to a substantial improvement (3.9% and 1.8% absolute WER reduction in the respective tasks) compared to simply using the KB and web texts.
  • The optimal operating point (best WER) was achieved with a smaller subset of selected web texts, indicating that the filtering process effectively identified high-quality data.
  • The performance was comparable to using three million domain-specific sentences, demonstrating the method's efficiency.

Limitations and Future Work

  • Threshold Determination: The method requires a development set (hundreds of sentences) to determine the optimal threshold ($\theta$) for sentence selection. While this is much smaller than full training data, it's still a requirement.
  • Reliance on Existing Corpora: The effectiveness hinges on the availability of a suitable, large, existing spoken dialogue corpus from a different domain for style matching.
  • Future Work: The authors suggest that this approach can be extended to other domains and potentially integrated into more complex dialogue systems, such as question-answering systems that rely on retrieved documents. They also highlight the potential for further refinement of the sentence selection criteria.

Summary updated

BibTeX
@inproceedings{misu2006bootstrapping,
  author = {Teruhisa Misu and Tatsuya Kawahara},
  title = {{A bootstrapping approach for developing language model of new spoken dialogue systems by selecting web texts}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure