Turn taking-based conversation detection by using DOA estimation

Yohei Kawaguchi, Masahito Togami, Yasunari ObuchiInterspeech 2010

TL;DR

This paper introduces a novel method for detecting conversation groups in real-world scenarios using only microphone arrays, eliminating the need for wearable microphones. By integrating Direction of Arrival (DOA) estimation with turn-taking rules, the system can accurately identify who is talking to whom, even in multi-party conversations.

What Problem Does This Paper Address?

Automatic analysis of conversations is crucial for applications like indexing recorded speech, enabling human-symbiotic robots, and understanding social interactions. While much research has focused on "planned conversations" (e.g., meetings), this paper tackles the more challenging domain of "unplanned conversations" (e.g., spontaneous encounters). A key challenge in unplanned conversations is identifying distinct conversation groups when multiple conversations are happening simultaneously. Existing methods often rely on wearable microphones, which are impractical for spontaneous interactions. This paper aims to bridge this gap by developing a system that can detect these conversation groups using only fixed microphone arrays.

Methods

The proposed method combines two main components:

  1. Direction of Arrival (DOA) Estimation-based Voice Activity Detection (VAD): Instead of relying on individual microphones, the system uses a microphone array to estimate the direction from which speech is coming. A Modified Delay-and-Sum Beamformer (MDSBF) is used to estimate the DOA for each time-frequency bin. This DOA information is then used to create a DOA histogram, which approximates the power of each speaker's voice. A thresholding operation on a calculated signal-to-noise ratio (SNR) is then applied to determine voice activity for each estimated speaker direction.
  2. Turn-Taking Rules for Conversation Detection and Speaker Count Estimation: The core idea is that speakers within the same conversation tend to follow turn-taking rules (minimizing gaps and overlaps). The paper leverages this by calculating the mutual information between the voice activities of different estimated speaker directions. High mutual information suggests a strong correlation, which is interpreted as speakers belonging to the same conversation. Conversely, low mutual information indicates speakers are likely in different conversations. This mutual information is also used to estimate the number of speakers, a crucial parameter for the k-means clustering used in DOA estimation. The system iteratively refines the speaker count by decreasing it and re-evaluating the turn-taking criteria until a stable state is reached. Finally, agglomerative clustering based on the mutual information is used to form conversation groups.

Datasets

The experiments were conducted using recorded casual conversations with 4 speakers and an 8-microphone array. The data was split into 4 sessions, each 60 seconds long, with different speaker configurations to simulate various scenarios. No new datasets were introduced; existing recorded data was used for evaluation.

Key Insights and Ideas

  • Wearable-free conversation detection: The most significant insight is that robust conversation detection can be achieved without requiring participants to wear microphones, making it suitable for spontaneous, real-world scenarios.
  • Turn-taking as a unifying principle: The paper demonstrates that turn-taking rules, a fundamental aspect of human conversation, can be effectively used not only for detecting conversation groups but also for estimating the number of speakers. This is a clever way to leverage inherent conversational structure.
  • DOA estimation as a proxy for speaker identity: By estimating the direction of arrival, the system can differentiate between speakers and track their voice activity, even in the presence of overlapping speech and reverberation.
  • Mutual information for correlation: Using mutual information as a metric to quantify the inverse correlation of voice activity between speaker directions is a sound statistical approach for identifying conversational partners.

Key Results

  • Conversation Detection Accuracy: The proposed method achieved an accuracy of 83% for conversation detection.
  • Speaker Count Estimation Accuracy: The accuracy for estimating the number of speakers was 96%.
  • Performance Comparison: The paper claims that the performance of their microphone-array-only method is comparable to conventional methods that rely on wearable microphones.
  • Window Size Impact: The accuracy of both conversation detection and speaker count estimation was found to be highly dependent on the window size used for calculating the turn-taking criteria. Longer window sizes (e.g., 60 seconds) yielded better results, suggesting the method is more effective for longer conversations.

Limitations and Future Work

  • Short Conversations: The method's performance degrades for shorter conversations (around 10 seconds or less) because the inverse correlation between speakers cannot be reliably detected. The authors suggest combining their method with conventional approaches based on auditory transfer characteristics for such cases.
  • Speaker Changes: The paper acknowledges that the method might incorrectly group speakers if a speaker switches conversations.
  • More Than Two Conversations: The current experimental evaluation primarily focused on scenarios with two simultaneous conversations. Future work should involve evaluating the method with more than two concurrent conversations and for more complex speaker configurations.
  • Speaker Switching: Investigating how the system handles situations where a speaker transitions between different conversations is also a direction for future research.

Summary updated

BibTeX
@inproceedings{kawaguchi2010turn,
  author = {Yohei Kawaguchi and Masahito Togami and Yasunari Obuchi},
  title = {{Turn taking-based conversation detection by using DOA estimation}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure