Floor holder detection and end of speaker turn prediction in meetings

Alfred Dielmann, Giulia Garau, Hervé BourlardInterspeech 2010

TL;DR

This paper introduces novel, fully automatic methods for identifying who is speaking in a meeting and predicting when they will finish their turn. By analyzing non-verbal cues and using advanced machine learning models, they achieve state-of-the-art performance in both floor holder detection and end-of-turn prediction, crucial for more natural human-computer interaction and communication analysis.

What Problem Does This Paper Address?

Conversations, especially in multi-party settings like meetings, are complex dynamic systems. Understanding who has the "floor" (i.e., who is currently speaking and expected to continue) and when they might yield it is fundamental to smooth communication. This ability is vital for applications ranging from intelligent dialogue systems and embodied conversational agents that need to engage naturally, to tools for automatic summarization, topic detection, and even "virtual video directors" for teleconferencing. While previous work has focused on dyadic conversations or relied heavily on manual annotations (like speech transcripts, gestures, or dialogue acts), this paper tackles the more challenging problem of analyzing multi-party meetings automatically, using only non-verbal cues and without requiring human-labeled data for feature extraction.

Methods

The authors propose two main approaches:

  • Unsupervised Floor Holder Detection and End-of-Speaker-Turn Detection: This method leverages existing outputs from speaker diarization (identifying who spoke when) and speech activity detection (identifying speech segments). These fine-grained segments are then post-processed using probabilistic models and cross-correlation techniques to group them into coherent speaker turns. Two specific techniques, Probabilistic Segment Filtering and Joint Maximum Cross-correlation Filtering (JMXC), are used to refine these segments and identify the active speaker. This approach is designed for offline analysis.
  • Supervised Joint Floor Holder Detection and End-of-Speaker-Turn Prediction: This approach is geared towards online applications and uses a Dynamic Bayesian Network (DBN). It models a combination of multimodal features extracted every 30ms, including:
  • Prosodic Features: Pitch contour, energy, and speaking rate.
  • Overlapping Speech Features: Derived from cross-correlation analysis of audio signals.
  • Visual Features: Motion intensity and center of motion from video data. These features are integrated early on into a single observable vector. The DBN then predicts the sequence of floor holders and the likelihood of a turn ending at each time step. A key innovation here is the joint prediction of both floor holder and turn end, and the use of automatically extracted features.

Datasets

The primary dataset used for both training and evaluation is the AMI meeting corpus. Specifically, the authors utilized the scenario subset, which comprises 138 meetings (72 hours) of multi-party conversations involving four participants. The corpus is rich in multimodal data, including headset microphones, distant microphones, and video cameras. For evaluation, they used 12 manually annotated meetings for unsupervised experiments and 123 AMI scenario meetings for training and 12 manually annotated meetings for testing the supervised DBN model. They also demonstrate that Dialogue Act (DA) annotations can serve as a reliable proxy for manual floor holder annotations, facilitating supervised learning.

Key Insights and Ideas

  • Non-Verbal Cues are Sufficient: The research strongly suggests that non-verbal cues (prosody, visual motion, overlapping speech) are powerful enough for both floor holder detection and end-of-turn prediction, reducing reliance on speech content.
  • Joint Prediction is Powerful: Simultaneously predicting who holds the floor and when their turn ends, rather than treating them as separate tasks, leads to better overall performance.
  • Dialogue Acts as a Proxy: The finding that Dialogue Act annotations can reliably substitute for manual floor holder annotations is a significant practical contribution, enabling supervised learning on a larger scale.
  • Multimodal Features Enhance Performance: Combining prosodic and visual features, particularly with the DBN, yields superior results compared to using individual feature sets or unsupervised methods.
  • Challenging Recording Setup: The successful application of their methods to data from Multiple Distant Microphones (MDM) is noteworthy, as this setup is less intrusive and more portable than using individual headset microphones.

Key Results

  • Unsupervised: The best unsupervised setup (IHM Speech Activity Detection + JMXC filtering) achieved a Floor Error Rate (FER) of 16.9% and an end-of-speaker-turn F1-score of 0.45.
  • Supervised: The DBN model with prosodic features (pitch, energy, ROS) achieved the best performance, with a Floor Error Rate of 13.2% and an end-of-speaker-turn F1-score of 0.50. This F1-score is competitive with, and slightly outperforms, the state-of-the-art reported in prior work (0.48).
  • The supervised DBN model runs at more than twice the real-time speed on a single core processor.

Limitations and Future Work

  • Limitations:
    • The unsupervised approach using IHM Speech Activity Detection requires access to individual headset microphones and prior knowledge of the number of speakers, making it more constrained.
  • While the DBN model is fast, it relies on supervised training, which requires annotated data.
  • Future Work: The authors suggest that further improvements could be made by exploring more sophisticated DBN models or other machine learning architectures. Investigating the impact of different feature integration strategies and expanding the analysis to more diverse meeting scenarios and cultural contexts would also be valuable. The potential for using these techniques in real-time interactive systems is a clear avenue for future research.

Summary updated

BibTeX
@inproceedings{dielmann2010floor,
  author = {Alfred Dielmann and Giulia Garau and Hervé Bourlard},
  title = {{Floor holder detection and end of speaker turn prediction in meetings}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure