Boosting Team Modeling through Tempo-Relational Representation Learning

Vincenzo Marco De Luca, Giovanna Varni, Andrea Passerini

TL;DR

This paper introduces TReNN and MT-TReNN, neural architectures that model human team dynamics by simultaneously capturing "who is interacting with whom" (relational) and "how interactions evolve" (temporal). By using multi-task learning to predict leadership and teamwork constructs, the authors achieve state-of-the-art performance while reducing computational overhead by up to 85% compared to single-task models.

What Problem Does This Paper Address?

Modeling how human teams function is a "holy grail" at the intersection of AI and Social Science. While social scientists have long viewed teams as Adaptive Complex Systems—where interactions over time create "emergent states" like leadership—most AI models are surprisingly reductive. They often treat team members as independent data points (static models) or ignore the network structure of their interactions.

Furthermore, practical applications like Human-Machine Teaming (HMT) require models that are not just accurate, but explainable. A surgical team or emergency response unit doesn't just need a "performance score"; they need to know why a certain dynamic emerged and how to improve it. This paper bridges the gap between complex social theories and scalable, interpretable deep learning.

Methods

The authors propose the Tempo-Relational Neural Network (TReNN) and its multi-task evolution, MT-TReNN. The architecture follows a "Space-Then-Time" (STT) strategy:

  1. Automatic Temporal Graph Extractor: Converts raw behavioral data (like who is speaking and when) into a sequence of graphs where nodes are people and edges represent interactions.
  2. Relational Learning (The "Space"): Uses Graph Convolutional Networks (GCNs) to encode the social structure at each specific moment, capturing how one person's behavior influences their neighbors.
  3. Temporal Learning (The "Time"): Uses Multi-Head Attention (MHA) to link these snapshots over time, allowing the model to capture long-range dependencies in team behavior.
  4. Multi-Task Decoder: Instead of training separate models for leadership and teamwork, MT-TReNN uses a shared "Social Embedding" backbone with specialized heads.
  5. Explainability Modules:
    • Factual: Uses Saliency Maps to show which person or interaction contributed most to a prediction.
  6. Counterfactual (CoDy): Suggests specific changes (e.g., "If this person hadn't interrupted here...") that would have led to a better teamwork score.

Datasets

The models were evaluated on two gold-standard (though relatively small) datasets of team interactions:

  • PAVIS: Audio-video recordings of 16 teams (4 people each) performing a "Winter Survival" decision-making task.
  • ELEA: 40 teams (3-4 people each) performing similar survival tasks, annotated for leadership and personality traits.
  • Annotations: Both datasets include high-quality labels from independent raters on constructs like Emergent Leadership, Leadership Style, and the "Big Five" of Teamwork (Adaptability, Back-up Behavior, etc.).

Key Insights and Ideas

  • The Power of Joint Modeling: The biggest "aha" moment is that models using only temporal data or only relational data consistently underperform. Teamwork is inherently "tempo-relational"—you cannot understand a leader without seeing how the group responds to them over time.
  • Social Embeddings: By training on multiple tasks (Leadership + Teamwork) simultaneously, the model learns a "Social Embedding" that is more robust and generalizable than task-specific features.
  • Efficiency via Multi-Tasking: MT-TReNN isn't just better; it's leaner. Because it shares a backbone across tasks, it uses 70% fewer parameters and achieves an 85% reduction in inference time compared to running multiple single-task TReNNs.
  • Actionable AI: The inclusion of counterfactual explanations moves the model from a "passive observer" to a "decision support tool," capable of providing specific feedback to improve team performance.

Key Results

  • Superior Accuracy: TReNN/MT-TReNN significantly outperformed baselines (Random Forests, LSTMs, and standard GCNs) across almost all metrics for Emergent Leadership and Teamwork components.
  • Leadership Prediction: On the PAVIS dataset, the model achieved an Accuracy@1 of ~86% for identifying the emergent leader, a significant jump over static or purely temporal models.
  • Computational Gains: MT-TReNN proved to be the fastest to train and the most memory-efficient, making it the only viable candidate for real-time monitoring in high-stakes environments like operating rooms.

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that PAVIS and ELEA are small. Deep learning typically thrives on "big data," which is notoriously hard to collect and annotate in social psychology.
  • Feature Narrowness: The current implementation relies heavily on paralinguistic features (speech patterns). Future work could incorporate body language, facial expressions, or physiological data (heart rate, etc.).
  • Ground Truth for XAI: While the explanations (Saliency/CoDy) look promising and align with human intuition, there is currently no objective "ground truth" to mathematically prove the explanations are correct.
  • Real-World Deployment: The model has yet to be tested in a live "Human-Machine Teaming" loop where the AI provides real-time feedback to humans.

Summary updated

Abstract

Team modeling remains a fundamental challenge at the intersection of Artificial Intelligence and Social Sciences. Although a variety of computational models have been proposed in the last two decades, most fail to integrate Social Sciences insights, such as the critical role of temporal interactions in shaping team dynamics, and do not meet key practical requirements for real-world applications, including the ability to provide real-time, actionable recommendations to enhance team performance. To address these limitations, in this paper, we propose a novel tempo-relational neural architecture that jointly models interactions between team members and the evolution of team dynamics through temporal graphs. We additionally propose a multi-task extension of the architecture that learns shared social embeddings for team members enabling the simultaneous prediction of multiple team constructs (e.g., Emergent Leadership, Leadership Style, and Teamwork components). Experiments on two state-of-the-art team datasets show that our tempo-relational architecture out performs temporal-only and relational-only approaches for team performance prediction, and that its multi-task extension substantially reduces training and inference time without loss of predictive performance. Finally, the integration of explainability techniques within the proposed architectures provides interpretable insights and actionable recommendations to support team improvement. These strengths make our approach particularly well-suited for human-centered artificial intelligence applications, such as intelligent decision-support systems in high-stakes collaborative environments.
BibTeX
@article{luca2026boosting,
  author = {Vincenzo Marco De Luca and Giovanna Varni and Andrea Passerini},
  title = {{Boosting Team Modeling through Tempo-Relational Representation Learning}},
  journal = {Cogn. Comput.},
  year = {2026},
  doi = {10.1007/s12559-026-10581-y},
}
Figure