Using maximum entropy (ME) model to incorporate gesture cues for SU detection

Lei Chen, Mary P. Harper, Zhongqiang Huang2006 · DOI

TL;DR

This paper proposes a Maximum Entropy (ME) model to improve sentence unit (SU) detection in spontaneous speech by integrating gestural cues with lexical and prosodic information. By framing SU detection as a posterior probability estimation problem and leveraging the temporal relationship between gestures and speech, their ME model significantly outperforms traditional HMM-based approaches, demonstrating the value of multimodal fusion for this task.

What Problem Does This Paper Address?

Accurately identifying sentence units (SUs) in spontaneous speech is crucial for improving downstream tasks like speech recognition and parsing. While previous work has explored using lexical and prosodic cues, humans naturally use multimodal information, including gestures, to communicate. Integrating gestural cues into SU detection models has been challenging. Traditional Hidden Markov Models (HMMs) optimize for joint probabilities, which don't directly align with SU classification error. Furthermore, gestures are not always present, and their absence shouldn't necessarily imply the absence of an SU boundary; it's the co-timing of gestures with speech that's informative. This paper aims to address these limitations by developing a more effective multimodal SU detection model.

Methods

The core contribution of this paper is the development and evaluation of a Maximum Entropy (ME) model for multimodal SU detection. Unlike generative HMMs, ME models directly estimate the posterior probability of an event (an SU boundary) given observations, which is more aligned with classification error.

The ME model integrates three types of features:

  • Lexical and Prosodic Features: These are modeled using a pre-existing HMM speech model. The posterior probabilities from this HMM are binned to create indicator functions for the ME model.
  • Gestural Features: Since automatic gesture feature extraction was not available for their dataset, the authors devised a novel approach based on the temporal relationship between gestures and speech. They analyzed the frequency of SUs in relation to words that are:
  • Inside a gesture.
  • Around a gesture boundary (before or after).
  • Within a gap between two gestures. This analysis informed the creation of indicator functions that capture the contextual relevance of gestures to SU boundaries.

The ME model combines these features by summing weighted indicator functions, where the weights ($\lambda_k$) are learned to maximize the conditional likelihood of the data. The model makes independent decisions for each inter-word boundary.

Datasets

  • VACE Multimodal Meeting Corpus: This is the primary dataset used for training and evaluation. It is described as significantly larger and more diverse than previous datasets used for multimodal SU detection, featuring 14 participants, 24,566 words, and 3,170 SUs. The corpus includes time-synchronized audio, video, and motion data, with manual transcriptions and gesture annotations.
  • RT04S Conversational Dialog Data Set: This dataset was used for training the initial language model (LMCTS) and prosody model (PMCTS) components, as it provided a larger annotated corpus for these specific modalities.

Key Insights and Ideas

  • Posterior Probability is Key: Framing SU detection as estimating the posterior probability $P(E|W, F, G)$ (event given words, prosody, and gesture) is more effective than maximizing joint probabilities used in traditional HMMs.
  • Gesture Timing Matters: The presence of gestures isn't as informative as their temporal relationship with speech. Gestures often terminate when a speaker stops talking, and their start/end can signal the completion of a thought unit (SU). Gaps between gestures also seem to be important for SU completion.
  • Context is Crucial for Gestures: Gestural features alone are weak predictors of SU boundaries. They become significantly more powerful when integrated within a model that already incorporates rich contextual information from speech (lexical and prosodic features).
  • ME for Multimodal Fusion: The Maximum Entropy framework provides a flexible and effective way to combine diverse multimodal features, especially when dealing with the conditional nature of gestural information.

Key Results

  • The proposed ME model, {LMCTS + PMCTS}ME+GMME, achieved the lowest Error Rate (ERR) of 36.94% on the VACE meeting data.
  • This represents a significant reduction in error compared to the baseline HMM speech model (LMCTS + PMCTS with an ERR of 38.36%), with a statistical significance of $p = 0.0175$.
  • The ME model also showed improvements in Insertion Rate (INS) and Deletion Rate (DEL) compared to the HMM baseline.
  • An ME model using only gestural features was found to be very weak, highlighting the importance of speech features for context.
  • Combining binned posteriors from the HMM speech model (including contextual information from adjacent boundaries) with gestural features yielded the best results.

Limitations and Future Work

  • Automatic Gesture Feature Extraction: The study acknowledges that the gestural features used were derived from human annotations of gesture segments, not automatically extracted. The authors note that automatically extracted features (like hold and effort) were not available for their VACE dataset.
  • Gesture Dynamics: The current ME model does not exploit the dynamics of gestures (e.g., speed, acceleration), which the authors believe could further improve prediction.
  • Future Work:
    • Investigate incorporating lexical and prosodic features directly into the ME model, rather than relying on HMM posteriors.
  • Expand the multimodal meeting corpus.
  • Identify additional gesture constraints.
  • Utilize automatically derived gesture features in future investigations.
  • Explore the impact of gesture dynamics.

Summary updated

Abstract

Accurate identification of sentence units (SUs) in spontaneous speech has been found to improve the accuracy of speech recognition, as well as downstream applications such as parsing. In recent multimodal investigations, gestur]al features were utilized, in addition to lexical and prosodic cues from the speech channel, for detecting SUs in conversational interactions using a hidden Markov model (HMM) approach. Although this approach is computationally efficient and provides a convenient way to modularize the knowledge sources, it has two drawbacks for our SU task. First, standard HMM training methods maximize the joint probability of observations and hidden events, as opposed to the posterior probability of a hidden event given observations, a criterion more closely related to SU classification error. A second challenge for integrating gestural features is that their absence sanctions neither SU events nor non-events; it is only the co-timing of gestures with the speech channel that should impact our model. To address these problems, a Maximum Entropy (ME) model is used to combine multimodal cues for SU estimation. Experiments carried out on VACE multi-party meetings confirm that the ME modeling approach provides a solid framework for multimodal integration.
BibTeX
@misc{chen2006using,
  author = {Lei Chen and Mary P. Harper and Zhongqiang Huang},
  title = {{Using maximum entropy (ME) model to incorporate gesture cues for SU detection}},
  year = {2006},
  doi = {10.1145/1180995.1181035},
}
Figure