A dual channel coupled decoder for fillers and feedback

Daniel Neiberg, Joakim GustafsonInterspeech 2011

TL;DR

This paper introduces a novel dual-channel decoder that models the interactions between speakers in a conversation to improve the detection of "fillers" (like "um" and "uh") and "feedback" (like "uh-huh" and "I see"). By jointly analyzing both speakers' audio, the system achieves over a 100% relative improvement in detecting these conversational cues compared to previous methods, making it useful for semi-automatic annotation of spoken dialogues.

What Problem Does This Paper Address?

Conversational speech is a dynamic, interactive process where speakers constantly influence each other. Traditional speech processing systems often treat each speaker in isolation, which can lead to errors, especially in noisy or overlapping speech scenarios. This paper focuses on improving the detection of non-lexical speech elements like fillers and feedback. These elements, while not carrying semantic content, are crucial for understanding conversational flow, turn-taking, and speaker engagement. The challenge lies in accurately identifying these cues, particularly when they overlap with other speakers' speech or occur in rapid succession, and in a way that can be scaled for large-scale annotation.

Methods

The authors propose a "dual-channel coupled decoder" built upon Hidden Markov Models (HMMs). The core innovation is the "coupled" aspect, meaning the model explicitly considers the dependencies between the two speakers' audio streams.

Here's a breakdown of the key methodological advancements:

  • Dual-Channel Topology: Instead of separate models for each speaker, the system uses a joint model that captures interactions. This is achieved by forming a "Cartesian product" of the individual speaker models, effectively creating a larger state space that represents pairs of states from each speaker.
  • Joint Transition Matrix: A novel transition matrix is introduced that models the probability of transitioning between joint states (speaker A in state X, speaker B in state Y) based on the previous joint state. This allows the model to learn how speaker A's state transitions influence speaker B's, and vice-versa.
  • Coupled Acoustic Model Retraining: The acoustic models (which map audio features to states) are also retrained symmetrically across both channels. This ensures that the model learns to interpret features from both speakers in a unified way.
  • Feature Engineering:
    • RASTA-processed MFCCs: Standard Mel-Frequency Cepstral Coefficients are used, enhanced by RASTA processing to remove channel bias and spikes.
  • Joint Maximum Cross-Correlation (JMXC): This new feature captures the similarity between the two audio channels, normalized by the energy of the non-target speaker. It's proposed as an approximation for the distance to the microphone, helping to distinguish between direct speech and crosstalk.
  • Normalized Fundamental Frequency Cepstral Representation: This feature captures pitch variations, which are known correlates of fillers and feedback.
  • Increased Frame Rate: The analysis frame rate was increased from 10 ms to 50 ms. While this might seem counterintuitive for fine-grained detection, it reduces computational complexity and allows for longer duration modeling within each state, leading to improved performance.

Datasets

The study uses the DEAL corpus. This corpus consists of informal, face-to-face conversations recorded with 6 subjects (2 shopkeepers, 4 buyers) performing a shopping task. The data was recorded with one microphone per speaker, resulting in two channels. The original labels for non-lexical entities (fillers, back-channels, etc.) were collapsed into broader categories: SILENCE, SPEECH, FILLER, and FEEDBACK for this study.

Key Insights and Ideas

  • Inter-speaker dependencies are crucial: The most significant insight is that modeling how speakers interact is not just beneficial but essential for accurately detecting conversational cues like fillers and feedback. Ignoring these dependencies leads to significant errors, particularly in distinguishing between speakers and handling crosstalk.
  • Joint modeling improves performance: The dual-channel coupled approach, especially with symmetric retraining of acoustic models and transition matrices, consistently outperforms single-channel models. This highlights the power of a holistic view of the conversation.
  • Feature combination is key: The addition of the JMXC feature, which captures inter-channel similarity, provides a substantial boost in performance, particularly for fillers and feedback. This suggests that features directly modeling the interaction are highly informative.
  • Trade-off between frame rate and complexity: Increasing the frame rate to 50 ms, while reducing temporal resolution, paradoxically improves performance by simplifying the model and allowing for longer state durations. This is an important practical consideration for real-time or large-scale processing.
  • Potential for semi-automatic annotation: The achieved F-scores are high enough to make the decoder a valuable tool for automatically segmenting speech into broad categories (VAD) and identifying specific illocutionary acts like fillers and feedback, significantly aiding manual annotation efforts.

Key Results

The paper presents a series of experiments demonstrating the progressive improvement of their decoder:

  • Baseline (S): A single-channel decoder using MFCCs and FO correlates.
  • Single-channel with JMXC (CS): Adding the JMXC feature to the single-channel decoder yields a significant improvement in average F-score.
  • Dual-channel with joint transition matrix (CDT): Introducing the coupled transition matrix further boosts performance, especially for FILLER and FEEDBACK.
  • Dual-channel with coupled acoustic model retraining (CDTA): This final configuration, with symmetric retraining of both transition matrices and acoustic models, achieves the highest performance.

Specifically, the final CDTA configuration, with 144 Gaussians per state, achieved an average F-score of 86.9%. This represents a relative improvement of over 100% for FILLER and FEEDBACK compared to their previous published results. The F-scores for SILENCE (96.8%) and SPEECH (92.4%) are also very high, indicating the decoder's effectiveness as a Voice Activity Detector.

Limitations and Future Work

  • Computational Cost: The authors note that coupled HMMs require significant computational effort and memory resources, which led them to reduce feature dimensionality using PCA.
  • Limited number of speakers: The experiments were conducted on a relatively small number of speakers (6 subjects).
  • Future Work: The paper suggests that the current system is already useful for semi-automatic annotation of large conversational corpora. Further work could explore extending this to multi-party conversations, investigating different feature combinations, and potentially exploring end-to-end deep learning approaches for this task.

Summary updated

BibTeX
@inproceedings{neiberg2011dual,
  author = {Daniel Neiberg and Joakim Gustafson},
  title = {{A dual channel coupled decoder for fillers and feedback}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure