Gaze patterns in turn-taking

Catharine Oertel, Marcin Włodarczak, Jens Edlund, Petra Wagner, Joakim GustafsonInterspeech 2012

TL;DR

This paper investigates how eye gaze patterns change around speech turn transitions, specifically looking at overlaps versus silences. They found distinct gaze behaviors for speakers and listeners depending on whether a turn change involves an overlap or a gap, and also identified unique gaze patterns associated with backchannels. This work contributes to a more nuanced understanding of conversational coordination and has implications for creating more natural human-computer interaction.

What Problem Does This Paper Address?

Face-to-face conversation is a complex dance of turn-taking, and while we understand the basics, the subtle cues that manage this flow are still being uncovered. Eye gaze is a powerful non-verbal signal, known to play a role in managing attention, expressing intimacy, and coordinating turns. However, previous research, like Kendon's seminal work, often lacked the temporal resolution to capture the rapid dynamics of these interactions. This paper aims to bridge that gap by providing a fine-grained analysis of gaze patterns around different types of turn transitions, including overlaps and silences, and also examining the role of backchannels. The core question is: how do speakers' and listeners' eye movements differ when a turn change is smooth versus when it involves an interruption or a pause, and how does this relate to backchanneling?

Methods

The researchers analyzed video recordings of spontaneous Dutch dialogues. They employed a frame-by-frame annotation scheme to track gaze, classifying it as either "looking at the interlocutor's face" (g) or "looking away" (x). This annotation was performed manually over a 3-second window before and after key turn transition events. Crucially, they defined several categories of turn transitions based on the presence and duration of overlaps and silences (longer or shorter than 130 ms, a threshold for perceived silences/overlaps). They also marked backchannels. Statistical analysis involved comparing gaze patterns across these categories using two-tailed randomization tests to assess the significance of observed differences.

Datasets

The study utilized a subset of the IFADV corpus, a video corpus of spontaneous Dutch dialogues. This subset comprised seven dialogues, totaling approximately 105 minutes of conversation. The participants in these dialogues knew each other well, suggesting a level of established rapport.

Key Insights and Ideas

  • Gaze is highly sensitive to turn transition type: The way speakers and listeners look at each other is not uniform; it changes significantly depending on whether a turn change involves an overlap or a silence.
  • Overlaps and silences have distinct gaze signatures: Overlaps with speaker change show a rapid increase in mutual gaze shortly after the overlap begins, while silences with speaker change exhibit a more gradual increase in gaze that peaks later.
  • Backchannels elicit unique gaze behavior: Backchannels are associated with a substantial increase in the previous speaker's gaze towards their partner, and the incoming speaker tends to look away more when producing a backchannel in a non-overlapped position.
  • "Smooth" transitions are visually different: Short gaps (less than 130 ms) and overlaps without speaker change show some similarities to smoother turn changes, but the incoming speaker's gaze behavior after taking the turn differs.
  • Gaze cues precede events: The observed gaze shifts often begin well before the actual turn transition, suggesting that gaze is used proactively in turn negotiation.

Key Results

  • Overlaps with speaker change (OV with SC) and silences with speaker change (SIL with SC): Both show a pattern of the previous speaker looking at their partner and the incoming speaker looking away. However, the timing differs: OV with SC has a sharp increase in gaze post-overlap onset, while SIL with SC has a gradual increase peaking about 1 second after the silence onset.
  • Overlaps without speaker change (OV without SC): This category showed less clear patterns, possibly due to fewer data points. However, previous speakers still increased their partner-oriented gaze after the overlap onset.
  • Silences without speaker change (SIL without SC): Previous speakers began looking away up to two seconds before the silence onset, suggesting a negotiation of turn-taking even when the floor isn't explicitly passed.
  • Backchannels (OV with BACK, SIL with BACK): These were characterized by a significant increase in previous speaker's partner-oriented gaze, much more so than in speaker changes. Incoming speakers looked away more during backchannels.
  • No-gap no-overlap (No-GAP No-OV): This category showed a familiar pattern of silences with speaker change, but with a smaller decrease in incoming speaker's gaze.

Limitations and Future Work

The authors acknowledge that the IFADV corpus consists of spontaneous Dutch dialogues between people who know each other, which might limit generalizability to other languages or interaction types. They also note that while their time step is small (10 ms), transitions between frames can occur at any point relative to turn boundaries.

Future work could leverage the finding that backchannels are cued up to 2 seconds in advance for audiovisual synthesis. Additionally, a more fine-grained annotation of turn-internal silences could provide deeper insights into gaze patterns during pauses. The authors suggest that these gaze cues could be valuable for designing more responsive and natural conversational agents.

Summary updated

BibTeX
@inproceedings{oertel2012gaze,
  author = {Catharine Oertel and Marcin Włodarczak and Jens Edlund and Petra Wagner and Joakim Gustafson},
  title = {{Gaze patterns in turn-taking}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure