What Decisions Have You Made?: Automatic Decision Detection in Meeting Conversations

Pei-Yun Hsueh, Johanna D. MooreNAACL 2007

TL;DR

This paper tackles the challenge of automatically identifying decisions made during collaborative meetings, a crucial step for improving meeting record usability and knowledge management. They achieve this by developing machine learning models that classify dialogue acts and topic segments related to decisions, finding that a combination of lexical, prosodic, contextual, and topical features yields the best results.

What Problem Does This Paper Address?

In collaborative work, especially in meetings, decisions are a critical outcome. However, locating these decisions within lengthy meeting recordings or transcripts can be incredibly difficult. Current tools for browsing and re-using meeting data often lack the ability to pinpoint decision points, making it hard for participants (especially new ones) to quickly grasp what was decided. While some research has focused on identifying specific types of decisions like action items, this paper aims to automatically detect decision-making units more broadly within multiparty conversations. The ultimate goal is to support computer-assisted meeting tracking, understanding, and the development of robust group information management systems.

Methods

The authors approach this problem by formulating it as a classification task at two levels:

  • Dialogue Acts (DAs): They aim to identify individual utterances that are decision-making in nature (DM DAs).
  • Topic Segments: They also seek to identify broader segments of the conversation that are focused on decision-making (DM Segments).

To achieve this, they employ Maximum Entropy (MaxEnt) models, a standard machine learning technique for classification. The core of their contribution lies in the empirical analysis of various feature types to determine which are most effective for decision detection. They investigate four main categories of features:

  • Lexical Features: Based on the words used in the dialogue acts (unigrams).
  • Prosodic Features: Including duration, pause, speech rate, energy, and pitch.
  • Contextual Features: Such as speaker roles (e.g., Project Manager, Marketing Expert) and meeting types (e.g., kick-off, design).
  • Topical Features: Related to the current topic of discussion and the position within a topic segment.

They systematically combine these features to train and evaluate their MaxEnt models, performing a quantitative analysis of their relative importance.

Datasets

The study utilizes the AMI meeting corpus, specifically a set of 50 scenario-driven meetings. These meetings involve four participants playing distinct roles in a design team. The corpus provides:

  • Manual transcripts for all meetings.
  • Individual sound files recorded by close-talking microphones.
  • Cross-talking sound files from desktop microphones.

Crucially, this corpus has been annotated for:

  • Dialogue Acts: Segmented into individual utterances.
  • Decision Information: Annotated through a two-phase process to identify DM DAs.
  • Topic Segmentation: Segments of conversation related to specific topics.

This dataset, with its detailed annotations, serves as the benchmark for their experiments.

Key Insights and Ideas

  • Feature Combination is Key: The most significant finding is that combining multiple feature types (lexical, prosodic, contextual, and topical) dramatically improves performance over using any single feature type or just prosodic features alone.
  • Lexical Features are Powerful: Lexical features (the words used) are identified as the most predictive indicators for both detecting DM DAs and DM Segments when used individually.
  • Context Matters: Speaker roles and meeting types provide valuable contextual information that helps in identifying decisions. For instance, Project Managers are more likely to produce DM DAs, and certain meeting types are more decision-prone.
  • Topic Segments Aid Precision: While lexical features are strong overall, topical features are particularly important for improving the precision of DM Segment detection. This suggests that decisions are often clustered within specific topics.
  • Timing Ambiguity: Using a "lenient match" measure (allowing a window around the hypothesized decision point) yielded better results, indicating that precisely pinpointing the exact moment of a decision can be challenging and that some temporal flexibility is beneficial.

Key Results

  • DM DA Detection: The model combining all features (ALL) achieved 72% precision and 86% precision (exact match and lenient match respectively) for detecting DM DAs.
  • DM Segment Detection: The ALL model achieved 82% overall accuracy for detecting DM Segments.
  • Feature Importance: Lexical features were found to be the most predictive when used alone. The combination of all features significantly outperformed baseline models.
  • Contextual Features Trade-off: While contextual features improved precision for DM DAs, they sometimes reduced recall, suggesting a tailoring effect that might miss broader decision-making signals.

Limitations and Future Work

  • Annotation Complexity: The two-phase annotation process for DM DAs might have led annotators to select acts with different functional roles (e.g., stating the topic, providing arguments, indicating agreement). This heterogeneity could degrade model performance. Future work could involve developing models for specific functional roles of DM dialogue acts.
  • Human Intervention: Many of the features used require manual annotation (transcripts, dialogue act segmentations, topic segmentations). The authors express a desire to move towards automatically generated features (e.g., using Automatic Speech Recognition - ASR) to enable fully automated decision detection immediately after a meeting or even during a meeting.
  • Hierarchical Annotation: The paper suggests that a hierarchical annotation scheme, similar to that used for action items, could better accommodate the different aspects of decision-making discussions.
  • Broader Decision Detection: The ultimate goal is to identify decisions in a more general sense, potentially using a hierarchical approach to accommodate diverse decision-making processes.

Summary updated

BibTeX
@inproceedings{hsueh2007what,
  author = {Pei-Yun Hsueh and Johanna D. Moore},
  title = {{What Decisions Have You Made?: Automatic Decision Detection in Meeting Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure