Towards the Automatic Detection of Involvement in Conversation

Catharine Oertel, Céline De Looze, Stefan Scherer, Andreas Windmann, Petra Wagner, Nick CampbellCOST 2102 Conference 2010 · DOI

TL;DR

This paper explores how to automatically detect a speaker's level of "involvement" in a conversation using acoustic cues (voice pitch, intensity) and body/face movement. They propose that involvement is a continuous, scalar phenomenon rather than a binary state, and their findings suggest that these multimodal cues can reliably identify distinct levels of engagement, paving the way for more nuanced human-computer interaction.

What Problem Does This Paper Address?

Despite decades of research in speech technology, we still lack a deep understanding of how variations in speech and non-verbal cues contribute to human communication and social interaction. Simply mapping text to speech isn't enough to capture the rich social information exchanged. This paper tackles the challenge of automatically detecting "involvement," defined as the perceived personal importance or interest evoked by a stimulus within a specific situation. Understanding and quantifying involvement is crucial for building more natural and responsive human-machine interactions, moving beyond one-to-one mappings to incorporate social dynamics.

Methods

The researchers developed an annotation scheme to perceptually rate involvement on a 1-10 scale, though only levels 4-9 were used in practice. They then analyzed acoustic features, specifically:

  • Voice Level and Span: Measured by F0 median (mean pitch) and F0 max/min (pitch range), normalized by z-score.
  • Articulation Rate: Calculated as syllables per second, also z-score normalized.
  • Intensity: Measured by RMS (Root Mean Square) of the audio signal, z-score normalized.

Additionally, they incorporated a multimodal aspect by analyzing body and face movement. This involved extracting facial and body coordinates using the Viola Jones algorithm and calculating movement changes. Normalization was applied to account for speaker distance and body size. Statistical analyses, primarily ANOVA, were used to test their hypotheses.

Datasets

The study utilized the D64 corpus, a multimodal corpus of approximately 8 hours of conversation recorded over two days. The corpus consists of recordings from five participants engaging in undirected conversations covering various topics. The data was selected to include 1 hour of recordings for analysis.

Key Insights and Ideas

  • Involvement as a Scalar Phenomenon: Contrary to some previous work treating involvement as binary, this study strongly suggests it's a continuous, scalar phenomenon, meaning there are degrees of involvement rather than just "involved" or "not involved."
  • Multimodal Cues are Key: Both acoustic features (voice pitch, intensity) and non-verbal cues (body and face movement) are identified as reliable indicators of involvement.
  • Voice Span and Intensity as Strong Indicators: Higher involvement correlates with a wider pitch span and increased intensity.
  • Articulation Rate is Not a Reliable Indicator: Interestingly, the study found no significant correlation between articulation rate and involvement.
  • Movement Correlates with Involvement: Preliminary analysis of movement data shows a positive correlation with involvement, though this can be speaker-dependent.

Key Results

  • Voice Level and Span: Involvement level 6 was significantly higher than level 5, and level 8 was significantly higher than level 7. The F0-max/min (pitch range) also increased with involvement.
  • Intensity: A clear increasing trend was observed, with higher involvement levels showing significantly higher intensity.
  • Movement: For one speaker (F), movement increased with involvement. For another speaker (C), the trend was less consistent, potentially due to factors like holding a laptop.
  • Articulation Rate: No significant changes in articulation rate were observed across different involvement levels.

Limitations and Future Work

  • Limited Movement Data: Movement analysis was only conducted for two speakers, limiting the generalizability of these findings.
  • Speaker-Specific Movement Trends: The inconsistent movement patterns for speaker C highlight the need for further investigation into individual differences and contextual factors.
  • Annotation Granularity: While a 1-10 scale was chosen, only levels 4-9 were used, and the statistical significance of some adjacent levels was not always clear. The paper notes that the number of levels to quantify involvement is still an open question.
  • Future Work: The authors plan to build a statistical model that integrates both acoustic and movement data to create a more robust automatic measure of involvement. They aim to further clarify the mutual information between these cues and continue using a scalar approach for quantifying involvement. This work has potential applications in multimodal corpus search, automatic spoken dialog systems, robotics, and games.

Summary updated

BibTeX
@inproceedings{oertel2010towards,
  author = {Catharine Oertel and Céline De Looze and Stefan Scherer and Andreas Windmann and Petra Wagner and Nick Campbell},
  title = {{Towards the Automatic Detection of Involvement in Conversation}},
  booktitle = {COST 2102 Conference},
  year = {2010},
  doi = {10.1007/978-3-642-25775-9\_16},
}
Figure