Towards flexible representations for analysis of accommodation of temporal features in spontaneous dialogue speech

Spyros Kousidis, David Dorran, Ciaran McDonnell, Eugene CoyleInterspeech 2009

TL;DR

This paper proposes a new way to represent spontaneous human conversations, moving beyond traditional "turn-taking" models to better capture the nuances of temporal features like pauses and overlaps. This flexible representation allows for a deeper analysis of how speakers "accommodate" to each other in real-time, which is crucial for building more natural and human-like spoken dialogue systems.

What Problem Does This Paper Address?

Modern spoken dialogue systems (SDSs) aim to mimic human-like interaction, but current representations often fall short. Traditional approaches focus on discrete "turns" where one person speaks and then the other. However, real human conversations are far more fluid, characterized by synchrony, active listening, and smooth exchanges that include overlapping speech and non-interrupting interjections. These dynamic temporal features, crucial for natural dialogue, are not well-captured by simple turn-based models. The paper addresses this by proposing a new representation that can better analyze how speakers adjust their timing (pauses, overlaps) in response to each other, a phenomenon known as temporal accommodation. This is essential for SDSs to achieve a more "human metaphor" and enhance user experience.

Methods

The core of the paper's contribution is a novel representation schema that abandons the concept of "turns" in favor of "turn shares" and other temporal measures. Instead of analyzing discrete turns, the authors propose analyzing dialogue segments within fixed-length "frames." Within each frame, they calculate:

  • Active Time (AT): The proportion of time a speaker is actively producing speech (including non-speech vocalizations like sighs).
  • Passive Time (PT): The proportion of time a speaker is silent.
  • Turn Share (TS): The proportion of a speaker's active time within a frame relative to the total active time of all speakers.
  • Joint Active Time (JAT): The proportion of time in a frame where at least one speaker is active (1 - proportion of total silence). This indicates the overall "liveliness" of the dialogue segment.
  • Overlap Rate (OR): The frequency of a speaker starting to speak before the other has finished.
  • Average Pause Length (APL): The average duration of pauses attributed to a speaker.

This approach allows for a more continuous and nuanced analysis of temporal dynamics, moving away from the rigid structure of turn-taking.

Datasets

The paper uses a corpus of spontaneous dialogues recorded from cooperative task experiments. Interactants had audio-only contact in separate isolation booths. The specific corpus is described as being the same as used in a previous work by the authors [10]. No new datasets were introduced.

Key Insights and Ideas

  • Beyond Turns: The traditional turn-taking model is insufficient for spontaneous dialogue because it struggles with overlapping speech and non-interrupting interjections. A turn-share approach offers a more flexible and accurate representation.
  • Temporal Accommodation is Real: The analysis suggests that speakers do indeed adjust their temporal features (pauses, overlaps) based on the dialogue context, specifically in relation to their turn share and overall dialogue activity.
  • Dialogue Activity Matters: Higher dialogue activity (JAT) correlates with more frequent overlaps, suggesting speakers are more likely to interject when the conversation is lively.
  • Pause Behavior: Average pause length is negatively correlated with dialogue activity (JAT), meaning shorter pauses occur when the dialogue is more active.
  • Turn Share Context: While average pause length isn't directly correlated with turn share, it is weakly correlated with an "exchange rate" (ER) derived from turn shares, indicating that pauses are shorter when the dialogue is more balanced.

Key Results

The paper presents experimental results in Table 2, showing correlations between temporal features (APL, OR) and dialogue context measures (JAT, ER, TS) for several dialogue segments. Key findings include:

  • APL and JAT: Negative correlation (-0.5 to -0.7), meaning shorter pauses occur in more active dialogue segments.
  • OR and JAT: Positive correlation (0.3 to 0.5), meaning more overlaps occur in more active dialogue segments.
  • OR and TS: Weak negative correlation (-0.2 to -0.4), indicating speakers overlap less when they dominate the turn.
  • APL and ER: Negative correlation (-0.3 to -0.5), suggesting shorter pauses occur when the dialogue exchange rate is higher (more balanced turn shares).

All correlations reported are significant at the 95% level.

Limitations and Future Work

  • Information Loss: The "sliding window" averaging process inherently leads to a loss of temporal resolution. The frame length chosen involves a trade-off between resolution and the ability to capture broader trends.
  • Validation Needed: The authors emphasize that this is an exploratory study, and the results need to be validated with further experiments.
  • Future Directions: The proposed representation can be used to inform interaction management components in SDSs, allowing them to adapt thresholds and latencies based on JAT and TS. The authors suggest that this approach can be combined with low-level acoustic and prosodic features for a more comprehensive analysis. They also note that the overlap time (TO) could be a useful indicator for automatic recognition of activation in emotional speech.

Summary updated

BibTeX
@inproceedings{kousidis2009towards,
  author = {Spyros Kousidis and David Dorran and Ciaran McDonnell and Eugene Coyle},
  title = {{Towards flexible representations for analysis of accommodation of temporal features in spontaneous dialogue speech}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure