Class Model Adaptation for Speech Summarisation

Pierre Chatain, Edward Whittaker, Joanna Mrozinski, Sadaoki FuruiNAACL 2006

TL;DR

This paper tackles the challenge of automatically summarizing spoken language, which is inherently difficult due to speech recognition errors. The authors propose using "class models" to adapt linguistic models, significantly improving summarization accuracy and robustness, especially for spontaneous speech. This work demonstrates a practical approach to making speech summarization more reliable and effective.

What Problem Does This Paper Address?

Automatic speech summarization aims to condense spoken content into a shorter, informative summary. However, a major hurdle is the inherent inaccuracy of Automatic Speech Recognition (ASR) systems, particularly for spontaneous speech. Spontaneous speech is riddled with disfluencies, repetitions, and repairs, making it much harder to transcribe accurately than read text. These ASR errors propagate into the summarization process, degrading the quality of the generated summaries. Previous work has shown that adapting linguistic models (LiMs) can improve summarization. This paper extends that research by investigating the use of "class models" within the LiM adaptation framework, aiming to enhance robustness and performance across various evaluation metrics.

Methods

The core of the summarization system involves two steps: sentence extraction and sentence compaction. For this study, only the sentence extraction component was utilized, as compaction showed minimal impact on their experimental data.

The system scores each sentence based on a weighted combination of three factors:

  • Confidence Score (C(wi)): Derived from the ASR output, representing the probability of a word being correctly recognized.
  • Significance Score (I(wi)): Based on word frequencies from large corpora (WSJ and Brown), indicating how common or important a word might be.
  • Linguistic Score (L(wi)): Calculated from the adapted linguistic model, reflecting the likelihood of a word sequence.

The key innovation lies in the linguistic model adaptation using class models. Instead of directly adapting word-level models, they adapt models that group words into classes. This is particularly beneficial when dealing with sparse data, as class models can generalize better. The linguistic model probability for a word sequence is calculated by linearly interpolating probabilities from different models (Equation 2). For class-based models, this probability is further broken down into the probability of a word belonging to a class and the probability of that class appearing given a history of classes (Equation 3). They combine a baseline LiM (LiMB) with component LiMs trained on different data sources (e.g., hand-made summaries, conference proceedings) using this interpolation.

Datasets

  • Translanguage English Database (TED) corpus: Used for experiments. It consists of 9 spontaneous speech talks, each manually summarized by nine different humans for 10% and 30% summarization ratios.
  • Speech recognition transcriptions (ASR): Obtained for the TED talks, with an average word error rate of 33.3%.
  • Conference proceedings corpus: A large corpus (17.8M words) from speech and signal processing, used to generate the baseline LiM and word classes.
  • WSJ and Brown corpora: Used to calculate word frequencies for the significance score.
  • Hand-made summaries: Used to train component LiMs for topic adaptation.

Key Insights and Ideas

  • Class models offer superior robustness: For spontaneous speech, where ASR errors are common and data can be sparse, class models provide better generalization than word models. This is because they can leverage broader patterns rather than being overly sensitive to specific word errors.
  • Adaptation is crucial for spoken summaries: Simply using a baseline model is insufficient. Adapting the linguistic model, especially with class-based approaches, leads to significant improvements across multiple evaluation metrics.
  • Different metrics tell different stories: While SumACCY (word accuracy) showed consistent improvements, ROUGE metrics (which focus on n-gram overlap) sometimes showed less dramatic gains, highlighting the importance of using a suite of evaluation metrics.
  • Topic adaptation is beneficial: Using component LiMs trained on relevant domain data (like conference proceedings) helps the summarizer capture important information and generate summaries in a similar style to human-generated ones.

Key Results

The paper reports significant improvements when using class models for adaptation, particularly on ASR output:

  • SumACCY: Relative improvements ranged from 7.6% to 15.9% for the 10% summarization ratio and 11.5% to 12.9% for the 30% ratio.
  • ROUGE metrics: Relative improvements for Class and Mixed models ranged from 6.0% to 22.2% for the 10% summarization ratio and 7.4% to 20.0% for the 30% ratio.
  • These improvements were more pronounced than those achieved with word-only adaptation.

Limitations and Future Work

  • Data Sparsity: While class models help, the inherent sparsity of data, especially in spontaneous speech and for hand-made summaries, remains a challenge.
  • Difference between Class and Mixed: The paper notes that the "Mixed" experiment (combining class and word models) yielded results very similar to the "Class" experiment. This is attributed to the development phase assigning more weight to the class model component.
  • Future Work: The authors suggest that further investigation into different types of component LiMs and their combinations could lead to even better results. The robustness of class models suggests they are a promising direction for future research in spoken language processing.

Summary updated

BibTeX
@inproceedings{chatain2006class,
  author = {Pierre Chatain and Edward Whittaker and Joanna Mrozinski and Sadaoki Furui},
  title = {{Class Model Adaptation for Speech Summarisation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure