Multimodal speaker diarization using oriented optical flow histograms

Mary Tai Knox, Gerald FriedlandInterspeech 2010 · DOI

Abstract

Speaker diarization in natural meeting environments remains a significant challenge for automated systems. This paper introduces a multimodal diarization framework that supplements traditional audio Mel-Frequency Cepstral Coefficients (MFCCs) with video-based oriented optical flow histograms. By capturing speech-related body movements and gestures, the proposed system achieves a 13.9% relative improvement in Diarization Error Rate (DER) on the AMI corpus compared to an audio-only baseline. The approach demonstrates robustness by focusing on motion patterns rather than explicit face tracking, making it suitable for realistic scenarios where participants may turn away from the camera.

Introduction

Speaker diarization—the task of determining "who spoke when"—is a fundamental component of automated meeting analysis. While humans perform this task effortlessly, AI systems often struggle in "wild" environments where participants move freely, interrupt each other, and turn away from microphones or cameras. Traditional systems rely almost exclusively on audio features, which can reach a performance ceiling in noisy or overlapping speech conditions. This paper addresses these limitations by integrating video-based motion features that serve as proxies for speech activity.

Methods

The researchers integrated video features into the established ICSI multi-stream speaker diarization engine. The system uses a Hidden Markov Model (HMM) framework where each state is a Gaussian Mixture Model (GMM) representing a specific speaker.

Audio and Video Streams

  • Audio Stream: Standard Mel-Frequency Cepstral Coefficients (MFCCs) are used to represent the short-term power spectrum of the audio signal.
  • Video Stream: The system employs oriented optical flow histograms to estimate 2-D motion. Motion is binned into horizontal, vertical, and diagonal categories. To prioritize meaningful gestures over minor noise, the histograms are weighted by the log-magnitude of the motion and normalized relative to the total motion in the frame.

Multimodal Fusion

The audio and video streams are combined using a weighted log-likelihood approach. A parameter $\alpha$ controls the relative weight of the video stream, allowing the system to adjust its reliance on visual data based on its perceived reliability.

Flowchart of the multimodal diarization process

Experimental Setup

The study utilized a subset of the AMI (Augmented Multi-Party Interaction) corpus, consisting of 12 meetings, each approximately 27 minutes long. The meetings featured four participants in a room equipped with a central microphone array and multiple camera viewpoints, including closeup cameras for each individual.

Experimental setup in the AMI meeting room showing camera and microphone placement

Results and Discussion

The primary metric for evaluation was the Diarization Error Rate (DER). The multimodal system achieved a DER of 25.3%, compared to 29.4% for the audio-only baseline, representing a 13.9% relative improvement.

Key insights from the results include:

  • Gesturing as a Proxy: The success of the system confirms that body motion and gesturing are effective descriptors of speech activity.
  • Robustness: By using motion histograms rather than coordinate-based tracking, the system remains location-independent and robust to speaker movement.
  • Variable Gains: Performance improvements varied by meeting, with some sessions showing dramatic DER reductions (up to 64.8%), while others showed slight degradation, highlighting the ongoing challenge of optimal multimodal fusion.

Conclusion

This research demonstrates that oriented optical flow histograms provide a valuable, robust source of information for speaker diarization. By moving beyond face tracking to capture general speech-related motion, the system achieves significant performance gains in realistic meeting scenarios. Future work will explore the application of these features to far-field camera views and the integration of automated speech/non-speech detection.

Summary updated

BibTeX
@inproceedings{knox2010multimodal,
  author = {Mary Tai Knox and Gerald Friedland},
  title = {{Multimodal speaker diarization using oriented optical flow histograms}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-113},
}
Figure