Co-Speech Gesture Detection Through Multi-Phase Sequence Labeling

Esam Ghaleb, Ilya Burenko, Marlou Rasenberg, Wim Pouw, Peter Uhrig, Judith Holler, Ivan Toni, Aslı Özyürek, Raquel FernándezWACV 2024

TL;DR

This paper introduces a novel framework for detecting co-speech gestures by reframing the problem as multi-phase sequence labeling, moving beyond traditional binary classification. By modeling gestures as sequences of preparation, stroke, and retraction phases, their approach significantly improves gesture stroke detection and offers a more nuanced understanding of gesture dynamics in natural conversations.

What Problem Does This Paper Address?

Gestures are a crucial part of human communication, often accompanying speech to illustrate, emphasize, or direct. While gesture analysis is important for fields like human-computer interaction and sign language recognition, current methods for detecting co-speech gestures often fall short. Many approaches treat gesture detection as a simple binary classification problem (gesture vs. no gesture), failing to capture the inherent temporal structure of gestures. Gestures aren't just single events; they unfold over time in predictable phases: preparation (moving towards the gesture), stroke (the main communicative part), and retraction (returning to a neutral pose). Furthermore, much existing research focuses on isolated gestures in controlled settings, neglecting the dynamic, continuous nature of gestures in natural, face-to-face conversations. This paper aims to bridge this gap by developing a method that can detect these multi-phase gestures in real-world dialogue.

Methods

The core of this paper's contribution lies in its novel framework for multi-phase sequence labeling. Instead of a binary classification, they model gesture detection as a task of identifying the sequence of phases a gesture goes through.

Here's a breakdown of their approach:

  • Spatio-Temporal Graph Convolutional Networks (ST-GCNs): They begin by representing upper-body skeletal movements within short time windows as spatio-temporal graphs. ST-GCNs are used to embed these graphs, capturing the dynamic patterns of joint movements within each window.
  • Transformer Encoder: To capture longer-term temporal dependencies across these embedded windows, they employ a Transformer encoder. This allows the model to understand how gestures evolve over a sequence of frames, learning contextualized embeddings.
  • Conditional Random Fields (CRFs): Finally, to leverage the sequential nature of gesture phases, they use linear-chain CRFs for structured prediction. CRFs are well-suited for sequence labeling tasks as they can model dependencies between adjacent labels, allowing for more accurate prediction of the entire gesture phase sequence.

This multi-phase sequence labeling approach is contrasted with traditional binary classification and simple classification methods in their experiments.

Datasets

The authors utilize a dataset created by Rasenberg et al. [17] for their research. This dataset comprises:

  • 19 face-to-face task-oriented dialogues involving 38 different subjects.
  • 16 hours of recorded naturalistic dialogue footage.
  • The dialogues are structured as a referential game, where one participant describes a novel object to another, eliciting spontaneous co-speech gestures.
  • Crucially, gesture strokes were manually identified and segmented per frame with high inter-annotator agreement (89.2%), providing a reliable basis for creating their multi-phase labels.

While the full dataset is not publicly available due to privacy concerns, the authors make available the estimated body poses obtained via MMPose [19] to ensure reproducibility.

Key Insights and Ideas

  • Gestures are inherently sequential: The paper strongly advocates for treating gestures not as isolated events but as temporal sequences with distinct phases. This perspective is key to unlocking more accurate detection.
  • Multi-phase labeling is superior: By explicitly modeling preparation, stroke, and retraction, their approach significantly outperforms binary classification, especially in capturing the full gesture unit.
  • Transformer encoders enhance context: The inclusion of Transformer encoders proved beneficial, particularly for improving recall and IoU scores, suggesting their ability to capture long-range dependencies in gesture sequences.
  • Boundary phases are challenging: The research highlights that while stroke detection is relatively robust, identifying the preparation and retraction phases (the boundaries of a gesture unit) is significantly more difficult, a common challenge in gesture analysis.
  • Naturalistic data is crucial: The use of a dataset from spontaneous, face-to-face dialogues is a significant step beyond controlled, silent gesture datasets, making the findings more relevant to real-world applications.

Key Results

The paper presents strong quantitative results demonstrating the effectiveness of their proposed framework:

  • Gesture Stroke Detection: Their multi-phase sequence labeling model with a Transformer encoder achieved the best F1 score (58.4) and IoU (41.2) for detecting gesture strokes, outperforming all baseline models.
  • Gesture Unit Detection: For detecting entire gesture units (preparation, stroke, and retraction combined), their multi-phase sequence labeling approach also achieved the best F1 score (67.2) and IoU (50.6).
  • Ablation Studies: The results consistently showed that sequence labeling methods outperform classification approaches. The Transformer encoder generally improved recall and IoU scores, and its removal led to less consistent performance.
  • Phase-Specific Performance: While their model performed best on the stroke phase (F1 score of 58.4), it struggled more with boundary phases (preparation: 34.9 F1, retraction: 28.3 F1). The retraction phase, in particular, showed very low performance for the model without the Transformer encoder.

Limitations and Future Work

  • Boundary Phase Detection: The authors acknowledge that their models, like others, have more difficulty identifying the preparation and retraction phases compared to the stroke phase. They hypothesize that the intricate dependencies of features near these boundary phases contribute to this challenge.
  • Retraction Phase Performance: The multi-phase sequential model without the Transformer encoder showed particularly poor results for the retraction phase. The authors attribute this to the CRF's reliance on previous hidden states and observed embeddings without explicit temporal context, which the Transformer provides.
  • Future Work: The paper suggests improving the detection of boundary phases as a key area for future research. They also aim to further explore the role of the Transformer encoder in capturing the fine-grained dynamics of gesture phases.

Summary updated

BibTeX
@inproceedings{ghaleb2024co,
  author = {Esam Ghaleb and Ilya Burenko and Marlou Rasenberg and Wim Pouw and Peter Uhrig and Judith Holler and Ivan Toni and Aslı Özyürek and Raquel Fernández},
  title = {{Co-Speech Gesture Detection Through Multi-Phase Sequence Labeling}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure