Invited Talk: Recognition and Understanding of Meetings

Steve RenalsNAACL 2010

TL;DR

This paper introduces the AMI corpus, a large, multimodal dataset for meeting analysis, and details the development of techniques for automatically recognizing and interpreting human communication in meetings. The work highlights the challenges and successes in building systems that can transcribe, segment, and summarize meetings, paving the way for more intelligent meeting support tools.

What Problem Does This Paper Address?

Meetings are a ubiquitous part of professional life, yet the information exchanged within them is often poorly preserved. Traditional meeting records like minutes are labor-intensive and capture only a fraction of the content, missing crucial nuances like decision-making rationale and subjective attitudes. This paper addresses the challenge of automatically capturing, processing, and understanding the rich, multimodal communication that occurs in meetings, which involves multiple speakers, spontaneous conversation, and information conveyed through audio, video, and other signals. The goal is to unlock the potential of recorded meetings for better access, analysis, and support.

Methods

The core of the paper's methodology revolves around the AMI corpus, a meticulously collected and annotated multimodal dataset. The authors describe the instrumented meeting rooms used for data capture, which employed multiple cameras, microphone arrays, and other sensors to record meetings unobtrusively.

Key techniques discussed include:

  • Multimodal Capture and Annotation: The AMI corpus was built with a focus on synchronizing various data streams (audio, video, digital pens, whiteboards) and annotating them at multiple levels, including word-level transcriptions, dialogue acts, topic segmentation, and even non-verbal cues like gestures and head pose.
  • Automatic Speech Recognition (ASR): The paper details the development of ASR systems specifically for spontaneous, conversational speech in noisy meeting environments. This involves advanced techniques like microphone array beamforming, vocal tract length normalization, speaker adaptation, and discriminative training.
  • Meeting Segmentation: Methods for segmenting meetings into meaningful units are explored, including identifying speech vs. non-speech, individual utterances, dialogue acts, and broader "meeting phases" or group actions. Dynamic Bayesian Networks (DBNs) are highlighted for their ability to integrate multimodal features for this task.
  • Summarization: The paper focuses on extractive summarization, where the most relevant parts of a meeting are identified and concatenated. This approach leverages dialogue acts, speaker features, prosody, and discourse cues, often using reduced-dimension text representations.
  • Application Prototypes: The developed recognition and interpretation components are integrated into meeting browsers and content linking devices. These prototypes allow users to navigate meeting archives, search for information using automatically generated transcripts and summaries, and receive contextually relevant documents in real-time.

Datasets

The primary dataset discussed is the AMI Meeting Corpus.

  • AMI Meeting Corpus: This corpus contains over 100 hours of meeting recordings. It includes:
  • Synchronized multimodal data streams.
  • Manual and automatic annotations at various levels (word-level transcriptions, dialogue acts, topic segmentation, named entities, gestures, head pose, etc.).
  • A significant portion (70%) is based on a design scenario involving four participants, allowing for controlled evaluations. The remaining 30% consists of naturally occurring meetings.
  • It is publicly available under a Creative Commons license.

Key Insights and Ideas

  • Multimodality is Crucial: Understanding meetings requires going beyond just speech. Gestures, visual cues, and other modalities provide essential information that can improve recognition accuracy and enable richer interpretation.
  • The AMI Corpus is a Game-Changer: The creation of a large, well-annotated, multimodal corpus like AMI was a critical step for advancing research in meeting understanding. It provides a standardized benchmark for evaluating and comparing different approaches.
  • ASR in Meetings is "ASR-Complete": The paper acknowledges that transcribing spontaneous, multi-party meeting speech is an extremely difficult problem due to overlapping speech, reverberation, and conversational disfluencies.
  • Extractive Summarization is Effective: Automatically identifying and extracting key segments from meetings can provide valuable summaries, even when using imperfect ASR output.
  • Real-time Processing Enables New Applications: The development of low-latency ASR systems opens up possibilities for real-time meeting support tools, such as content linking and interactive browsing.
  • Interdisciplinarity is Key: Addressing the complexity of meeting understanding requires collaboration between signal processing, speech recognition, natural language processing, and human-computer interaction researchers.

Key Results

  • Meeting Segmentation: A multistream DBN approach significantly reduced the "action error rate" for group action recognition to around 13%.
  • Dialogue Act Recognition: Degradation when moving from manual to automatic transcriptions for dialogue act classification and segmentation was less than 10% absolute.
  • Speech Recognition (Microphone Array): The core microphone array system achieved a word error rate of about 40%, which was reduced to about 30% after adaptation and feature combination.
  • Speech Recognition (Close-talking Microphone): The equivalent close-talking microphone system had a baseline word error rate of about 35%, reduced to less than 25% after further passes.
  • Real-time System Performance: The core system runs about five times slower than real-time, and the full system is about fourteen times slower than real-time on current hardware. A low-latency real-time system achieved an error rate of about 41% for microphone array input.

Limitations and Future Work

  • Social Signals: The authors acknowledge that while their work has focused on lexical content and multimodal streams, the social signals in meetings (e.g., social role, group dynamics) are at least as important and represent a significant challenge for future development.
  • Unsupervised Adaptation: The reliance on annotated in-domain data is a major limitation. Developing unsupervised and adaptive algorithms that can function without extensive new data collection for each domain is a key future challenge.
  • Complexity of Real-World Meetings: While the scenario-based meetings in the AMI corpus allow for controlled evaluation, real-world meetings often have more complex histories and contexts that are harder to model.
  • ASR Error Impact: Despite efforts, ASR errors still impact downstream tasks like summarization, and users reported ASR transcripts as unsatisfactory. Further improvements in ASR accuracy are needed.
  • Scalability: The paper mentions the need to address challenges with larger groups or family "meetings," suggesting that current approaches may not scale directly.

Summary updated

BibTeX
@inproceedings{renals2010invited,
  author = {Steve Renals},
  title = {{Invited Talk: Recognition and Understanding of Meetings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure