Teamwork Quality Prediction Using Speech-Based Features

Martín Meza, Lara Gauder, Lautaro Estienne, Ricardo Barchi, Agustín Gravano, Pablo Riera, Luciana FerrerSMM 2023 · DOI

TL;DR

This paper introduces a novel protocol for annotating teamwork quality directly from speech, addressing the lack of suitable datasets. They developed a system that can automatically predict teamwork quality from conversations with promising accuracy, opening doors for real-time team performance analysis.

What Problem Does This Paper Address?

Effective teamwork is crucial for success in many fields, from software development to healthcare. While we intuitively understand what good teamwork looks like, objectively measuring it, especially in real-time, is challenging. Existing methods often rely on external observers or complex sensor data, which can be intrusive or impractical. This paper tackles the problem of automatically assessing teamwork quality by leveraging only the audio recordings of team interactions. The core challenge is the scarcity of annotated speech datasets that specifically capture teamwork quality, making it difficult to train and evaluate predictive models.

Methods

The authors developed a new annotation protocol to label teamwork quality from speech. They applied this protocol to a Spanish version of the Objects Games corpus, where pairs of participants collaborate on a task. The annotation process involved four raters who assessed various aspects of teamwork, including a general "teamwork quality" metric, specific teamwork components (like leadership and adaptability), and social behaviors (like conversation flow).

For automatic prediction, they employed Random Forest Regressors (RFRs). They extracted two main sets of features from the audio:

  1. Acoustic Features: Using the eGeMAPS specification, which captures a wide range of acoustic properties like pitch, energy, and spectral characteristics.
  2. Manual Annotations: Features derived from manually annotated data, such as word counts, speech duration, silence duration, and turn-taking patterns.

They experimented with various combinations of these features, including normalized versions, to see which best predicted the human-annotated teamwork quality.

Datasets

The primary dataset used is a subset of the UBA Games and Directions Corpus, which is a Spanish version of the Objects Games Corpus. This corpus consists of 14 dialogues where participants play a collaborative game. The authors annotated 196 audio samples from this corpus.

For hyperparameter tuning of their prediction models, they used the Columbia Games Corpus, an English version of the Objects Games Corpus.

Key Insights and Ideas

  • Speech as a Rich Source for Teamwork: The study demonstrates that speech alone contains sufficient information to predict teamwork quality, even without visual cues or physiological data.
  • Annotation Protocol Validation: The developed annotation protocol achieved moderate to high inter-rater agreement (ICC of 0.64), suggesting its reliability for capturing subjective teamwork assessments.
  • Task Duration's Influence: Teamwork quality annotation agreement significantly decreases for longer tasks (over 100 seconds). This suggests that teamwork is dynamic and can change within a single task, making it harder for raters to provide a consistent score.
  • Number of Turns as a Strong Predictor: Surprisingly, the total number of turns taken by participants emerged as the most powerful single feature for predicting teamwork quality, outperforming even the total task duration. This highlights the importance of interaction frequency.
  • Overfitting Concerns: When combining many features, the models showed signs of overfitting, likely due to the relatively small dataset size. This suggests that simpler models or more data might be needed for robust predictions.

Key Results

  • The annotation protocol achieved an Intraclass Correlation Coefficient (ICC) of 0.64 for teamwork quality, indicating moderate to high agreement among raters.
  • The automatic prediction system using speech-based features achieved a coefficient of determination (R²) of 0.56.
  • The best single feature for prediction was turn_total_cnt (total number of turns).
  • For tasks shorter than 100 seconds, the inter-rater agreement improved to 0.79.
  • When considering features not directly related to task duration, the best performance achieved an R² of 0.30.

Limitations and Future Work

  • Task Duration Impact: The authors acknowledge that agreement drops for longer tasks and suggest splitting longer audio samples into shorter segments for more accurate annotations in future work.
  • Dataset Size: The relatively small dataset size led to overfitting issues when using many features. The authors plan to explore automatic extraction of "task-duration-aware" features in future work.
  • Generalizability: The current findings are based on a specific game-based task. Future research should investigate the generalizability of these methods to different types of collaborative tasks and domains.
  • Feature Engineering: While turn_total_cnt was a strong predictor, exploring more sophisticated combinations of features or different machine learning models could further improve performance.

Summary updated

BibTeX
@inproceedings{meza2023teamwork,
  author = {Martín Meza and Lara Gauder and Lautaro Estienne and Ricardo Barchi and Agustín Gravano and Pablo Riera and Luciana Ferrer},
  title = {{Teamwork Quality Prediction Using Speech-Based Features}},
  booktitle = {SMM},
  year = {2023},
  doi = {10.21437/smm.2023-1},
}
Figure