M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention

Yiming Tang, Abrar Anwar, Jesse Thomason

TL;DR

This paper introduces M3PT, a novel transformer-based model that can simultaneously predict multiple social signals (like speaking status and bite timing) from various modalities (pose, gaze, speech) in multi-party conversations. By employing a unique blockwise attention mechanism, M3PT effectively captures complex temporal and inter-participant interactions, outperforming previous task-specific approaches.

What Problem Does This Paper Address?

Understanding human social interactions is crucial for applications ranging from affective computing to social robotics. In multi-party settings, such as group conversations or shared meals, social signals are not isolated events. Instead, they form a complex, interwoven dance where one person's actions influence others. Previous research often focused on predicting a single social signal using task-specific models. This paper tackles the challenge of building a single, unified model that can predict multiple social signals from multiple modalities across multiple participants simultaneously, accounting for their intricate temporal and inter-personal dependencies. This is particularly relevant for applications like robot-assisted feeding, where understanding the social dynamics of a meal is key to providing natural and helpful assistance.

Methods

The core of M3PT is a causal transformer architecture. To handle the diverse and continuous nature of social signals (e.g., gaze, body pose), the authors first employ a Vector Quantized Variational Autoencoder (VQ-VAE) to tokenize these signals into discrete representations. This allows the transformer to process them efficiently.

The key innovation lies in M3PT's person-aware and modality-aware blockwise attention masking. Unlike standard causal transformers that use a simple lower-triangular mask, M3PT's mask is structured to:

  • Respect temporal order: It only attends to past and present timesteps.
  • Enable inter-participant reasoning: It allows each participant's signals to attend to their own past signals and the signals of other participants in the current timestep.
  • Handle modality diversity: It accounts for different types of social signals.

This blockwise masking is crucial for learning the "choreography" of social signals in a multi-party setting. Additionally, a "right-shifted residual connection" is introduced to prevent information leakage during training, a common issue in unidirectional decoder architectures.

Datasets

The primary dataset used is the Human-Human Commensality Dataset (HHCD). This dataset captures triadic dining interactions, providing multimodal data such as speech, body pose, and gaze, along with annotations for bite events and speaking status. The authors repurpose this dataset for their multi-party, multimodal social signal prediction task.

Key Insights and Ideas

  • Unified prediction is powerful: A single model capable of predicting multiple social signals from multiple modalities significantly outperforms specialized models.
  • Inter-participant attention is key: The ability to attend to what others are doing is critical for predicting an individual's behavior in a social context.
  • Discretization is a practical necessity: For continuous signals like pose and gaze, tokenization via VQ-VAE is an effective way to make them compatible with transformer architectures.
  • Temporal chunking matters: The choice of segment length for tokenization is a delicate balance. Too long, and the VQ-VAE might collapse information; too short, and the transformer might struggle to capture longer temporal dependencies. The paper finds 3-second segments to be a good compromise.

Key Results

  • Multimodality is essential: Using all available modalities (gaze, head pose, body pose, speech) leads to the best performance for both bite timing and speaking status prediction. Removing any single modality causes a substantial performance drop.
  • Bite timing and speech are highly informative: The paper highlights that gaze, words, and speaking status are the most informative signals for predicting bite timing. For speaking status, bite timing is also a crucial predictor.
  • Temporal context: While longer temporal contexts (up to 36 seconds) are explored, the results suggest that for this specific task and model, reducing the context slightly can sometimes improve performance, possibly due to increased noise in longer sequences.
  • Segment length impact: Longer segment lengths (e.g., 18 seconds) lead to a significant collapse in prediction performance, indicating that the transformer needs shorter, more granular temporal chunks to effectively learn temporal structure.

Limitations and Future Work

  • Discrete vs. Continuous Prediction: M3PT currently predicts discrete social signals. The authors acknowledge that predicting continuous signals like body pose directly is challenging, and future work will explore methods for this.
  • Generalizability to Continuous Signals: While M3PT can process continuous signals after VQ-VAE tokenization, the goal is to predict continuous signals themselves. The current performance on predicting discrete pose tokens is still lacking, and it's unclear if interlocutor behavior is sufficient to predict precise body poses.
  • Ethical Considerations: The authors raise important ethical points regarding consent for tracking social signals, potential for manipulation, and the need to avoid bias in models used for sensitive applications like assistive feeding. They suggest explicit consent and anonymization as mitigation strategies.
  • Broader Social Signal Prediction: Future work aims to expand the prediction to a wider range of social signals beyond bite timing and speaking status.

Summary updated

Abstract

Understanding social signals in multi-party conversations is important for human-robot interaction and artificial social intelligence. Social signals include body pose, head pose, speech, and context-specific activities like acquiring and taking bites of food when dining. Past work in multi-party interaction tends to build task-specific models for predicting social signals. In this work, we address the challenge of predicting multimodal social signals in multi-party settings in a single model. We introduce M3PT, a causal transformer architecture with modality and temporal blockwise attention masking to simultaneously process multiple social cues across multiple participants and their temporal interactions. We train and evaluate M3PT on the Human-Human Commensality Dataset (HHCD), and demonstrate that using multiple modalities improves bite timing and speaking status prediction.
BibTeX
@inproceedings{tang2026m,
  author = {Yiming Tang and Abrar Anwar and Jesse Thomason},
  title = {{M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention}},
  booktitle = {Multimodality, Social Signals, Human-computer Interaction},
  year = {2026},
  url = {https://openreview.net/forum?id=6BvBJyEBvz},
}
Figure