A Skip-Chain Conditional Random Field for Ranking Meeting Utterances by Importance

Michel GalleyEMNLP 2006 · OpenReview

Abstract

We describe a probabilistic approach to content selection for meeting summarization. We use skip-chain Conditional Random Fields (CRF) to model non-local pragmatic dependencies between paired utterances such as Question-Answer that typically appear together in summaries, and show that these models outperform linear-chain CRFs and Bayesian models in the task. We also discuss different approaches for ranking all utterances in a sequence using CRFs. Our best performing system achieves 91.3% of human performance when evaluated with the Pyramid evaluation metric, which represents a 3.9% absolute increase compared to our most competitive non-sequential classifier.

Introduction

Summarizing multi-party meetings presents unique challenges compared to text-based summarization. Meetings are inherently messy, characterized by high word error rates in speech-to-text transcripts, a lack of formal punctuation, and frequent ungrammaticality. A critical structural element of meetings is the Adjacency Pair (AP)—pragmatic links such as Question-Answer or Offer-Accept.

In natural conversation, the components of an AP are often separated by interruptions or "backchannel" comments (e.g., "yeah", "uh-huh"), making it difficult for standard sequence models like Hidden Markov Models (HMMs) or linear-chain Conditional Random Fields (CRFs) to capture these vital dependencies. This paper introduces a skip-chain CRF approach to explicitly model these long-distance relationships to better rank utterance importance for summarization.

Methodology

The core of the proposed approach is the Skip-Chain CRF, an undirected graphical model that incorporates both local and non-local dependencies.

Graphical model of a skip-chain CRF showing linear-chain edges between adjacent utterances and skip-chain edges between non-adjacent but pragmatically linked utterances.
  • Linear-chain edges: These model local dependencies, capturing the likelihood of an utterance $t$ being important given the label of the preceding utterance $t-1$.
  • Skip-chain edges: These model non-local dependencies. When an Adjacency Pair relationship is detected between two distant utterances (e.g., a question and its corresponding answer), a direct edge is added between them in the graphical model.

Ranking with CRFs

While CRFs typically find the most likely sequence of labels, summarization requires ranking utterances to meet specific length constraints. The authors found that raw CRF "costs" were insufficient for ranking. They proposed a "norm-CRF" approach, which uses CRF-trained parameters to estimate local posterior probabilities. This method combines the discriminative power of CRFs with the ranking flexibility required for extractive summarization.

Feature Engineering

The model utilizes a comprehensive set of 217 features (selected from an initial 1,171), including lexical (N-grams, word counts), acoustic (pitch and energy), structural (meeting position), and discourse features (fillers and backchannel tokens).

Experimental Setup

The models were evaluated using the ICSI Meeting Corpus, consisting of 75 naturally occurring meetings. Human-annotated "dialog act" labels from the MRDA Corpus were used to train the Adjacency Pair classifier. Gold-standard "important" utterances were sourced from the Edinburgh Extractive Summaries.

Results and Discussion

The skip-chain CRF significantly outperformed baseline models. The "norm-CRF" ranking method proved essential for achieving high performance in the ranking task.

Performance comparison (Pyramid scores) of different models across different summary lengths (k). The skip-chain CRF with norm-CRF ranking achieves the best results.

Key findings include:

  • Superior Performance: The best-performing model (an order-2 skip-chain CRF) achieved 91.3% of human performance (Pyramid score of 0.554), outperforming the best non-sequential classifier by 3.9% absolute.
  • The "Orphan" Problem: By modeling APs, the skip-chain model ensures that related utterances (like a question and its answer) are treated as a unit, preventing disjointed summaries where an answer might appear without its context.
  • Robustness: The model remained effective even with noisy Automatic Speech Recognition (ASR) output, achieving a Pyramid score of 0.504 despite a 34.8% word error rate.

Conclusion

This work demonstrates that modeling non-local pragmatic dependencies through skip-chain CRFs significantly improves meeting summarization. By bridging the gaps created by conversational interruptions, the model more accurately identifies the core content of multi-party dialogues. Future work includes exploring more sophisticated probabilistic inference techniques to handle more complex conversation structures beyond simple adjacency pairs.

Summary updated

Abstract

We describe a probabilistic approach to content selection for meeting summarization. We use skipchain Conditional Random Fields (CRF) to model non-local pragmatic dependencies between paired utterances such as Question-Answer that typically appear together in summaries, and show that these models outperform linear-chain CRFs and Bayesian models in the task. We also discuss different approaches for ranking all utterances in a sequence using CRFs. Our best performing system achieves 91.3% of human performance when evaluated with the Pyramid evaluation metric, which represents a 3.9% absolute increase compared to our most competitive non-sequential classifier.
BibTeX
@inproceedings{galley2006skip,
  author = {Michel Galley},
  title = {{A Skip-Chain Conditional Random Field for Ranking Meeting Utterances by Importance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
  url = {https://openreview.net/forum?id=HkWfBMzOZH},
}
Figure