Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization

Manling Li, Lingyu Zhang, Heng Ji, Richard J. RadkeACL 2019 · OpenReview

Abstract: Transcripts of natural, multi-person meetings differ significantly from documents like news articles, which can make Natural Language Generation models for generating summaries unfocused. We develop an abstractive meeting summarizer from both videos and audios of meeting recordings. Specifically, we propose a multi-modal hierarchical attention across three levels: segment, utterance and word. To narrow down the focus into topically-relevant segments, we jointly model topic segmentation and summarization. In addition to traditional text features, we introduce new multi-modal features derived from visual focus of attention, based on the assumption that the utterance is more important if the speaker receives more attention. Experiments show that our model significantly outperforms the state-of-the-art with both BLEU and ROUGE measures.

Introduction

Summarizing a meeting is significantly more challenging than summarizing structured text like news articles. Meeting transcripts are notoriously messy, characterized by "ill-formed utterances," frequent interruptions, and complex multi-party chatter. Standard Natural Language Generation (NLG) models, such as the Pointer-Generator Network (PGN), often struggle with these long, rambling transcripts, failing to identify which segments of the conversation are truly salient.

Traditional extractive methods, which select key sentences from the text, often result in disjointed and "robotic" summaries. To address these limitations, this research introduces an abstractive system that generates natural-sounding summaries by leveraging non-textual cues—specifically, where participants are looking—and hierarchical topic segmentation to ensure the output remains topically focused.

Methods

The proposed architecture utilizes a multi-modal hierarchical attention mechanism operating across three distinct levels: topic segments, utterances, and words.

Overview of the multi-modal meeting summarization architecture.

Visual Focus of Attention (VFOA)

Using the OpenFace tool, the system analyzes meeting video to track each participant's head pose and eye gaze. This is based on a social assumption: an utterance is likely more important if the speaker is receiving a high degree of attention from the rest of the group. This "received attention" acts as a salience filter for the model.

Hierarchical Attention and Joint Modeling

Instead of processing the entire transcript as a flat sequence, the model mimics human cognitive behavior by first segmenting the meeting into logical topics. It then applies attention within those segments to identify key utterances and words. The system is trained to perform topic segmentation and summarization simultaneously. This multi-task approach helps the model better understand the underlying structure of the conversation.

Pointer-Generator Network Base

The model is built upon a PGN architecture, allowing it to both generate new words from a vocabulary and "copy" specific keywords (e.g., "remote control") directly from the transcript to ensure technical accuracy.

Experimental Setup

The study utilized the AMI Meeting Corpus, a benchmark dataset for multi-modal meeting analysis containing over 100 hours of meetings regarding a design project for a new television remote control. The data includes synchronized audio, video, and manually revised Automatic Speech Recognition (ASR) transcripts. For this study, 97 meetings were used for training, with 20 for development and 19 for testing.

Key Results

The multi-modal approach demonstrated substantial gains over both extractive and text-only abstractive baselines. Quantitatively, the proposed model achieved a ROUGE-1 score of 53.29, a significant improvement over the text-only PGN baseline of 36.75. Similarly, it reached a BLEU-1 score of 40.98, compared to the PGN's 37.89.

Comparison of different summarization approaches against the manual ground truth.

Qualitatively, the multi-modal model successfully captured specific technical keywords that text-only models often ignored. As shown in the comparison above, the summaries produced were more well-organized and natural than extractive methods, which frequently produced choppy or incomplete sentences. Furthermore, the inclusion of VFOA features improved topic segmentation accuracy from 57.74% to 60.11%, suggesting that visual cues are deeply tied to the logical flow of human conversation.

Conclusion

By integrating social signals like eye contact with hierarchical topic modeling, this research provides a more robust framework for summarizing complex multi-party interactions. Future work will focus on improving VFOA estimation accuracy and expanding the model to include other human signals, such as hand gestures and facial expressions, to further refine the understanding of speaker intent and participant feedback.

Summary updated

Abstract

Transcripts of natural, multi-person meetings differ significantly from documents like news articles, which can make Natural Language Generation models for generating summaries unfocused. We develop an abstractive meeting summarizer from both videos and audios of meeting recordings. Specifically, we propose a multi-modal hierarchical attention across three levels: segment, utterance and word. To narrow down the focus into topically-relevant segments, we jointly model topic segmentation and summarization. In addition to traditional text features, we introduce new multi-modal features derived from visual focus of attention, based on the assumption that the utterance is more important if the speaker receives more attention. Experiments show that our model significantly outperforms the state-of-the-art with both BLEU and ROUGE measures.
BibTeX
@inproceedings{li2019keep,
  author = {Manling Li and Lingyu Zhang and Heng Ji and Richard J. Radke},
  title = {{Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=wCEtYVrbNS},
}
Figure