Automated Team Discourse Annotation and Performance Prediction Using LSA

Melanie J. Martin, Peter W. FoltzNAACL 2004

TL;DR

This paper introduces Latent Semantic Analysis (LSA) as a method to automatically analyze team communication and predict team performance in simulated military missions. By analyzing the semantic content of verbal interactions, LSA can categorize discourse and, more importantly, accurately predict how well a team will perform, offering a valuable tool for training systems.

What Problem Does This Paper Address?

Complex military operations increasingly rely on teams, making effective team training crucial. However, manually analyzing team communication to understand cognitive processes and predict performance is time-consuming, costly, and subjective. This paper addresses the need for automated methods to analyze team discourse, specifically to categorize the type of communication and to directly predict team performance, thereby enabling more efficient and objective training systems.

Methods

The authors propose two main applications of Latent Semantic Analysis (LSA) for analyzing team discourse:

  1. Automatic Discourse Tagging: LSA is used to automatically assign predefined tags (e.g., acknowledgement, fact, uncertainty) to individual utterances (turns) within team conversations. This approach aims to replicate manual tagging but with significant time savings. They developed an algorithm that identifies "most similar" turns in other transcripts and uses their tags to predict the tags for a given turn. They also introduced an "LSA+" version that incorporates simple discourse features (like question marks) to improve accuracy.
  2. Team Performance Prediction:
    • Tag-based prediction: The frequencies of automatically generated tags are correlated with a composite team performance score.
  3. Transcript-based prediction: LSA is used to measure the semantic similarity between entire mission transcripts. The performance of a team is then estimated by averaging the performance scores of its most similar transcripts in the dataset.

Datasets

The primary dataset used is the UAV-Corpus, consisting of 67 transcripts from 11 teams completing 7 simulated Uninhabited Air Vehicle (UAV) missions. This corpus contains approximately 20,545 utterances and 232,000 words. The transcripts were manually annotated by three annotators using the Bowers Tag Set. For training LSA, an additional 2257 documents (training materials and interviews related to UAVs) were added, creating a corpus of 22,802 documents for a 300-dimensional semantic space.

Key Insights and Ideas

  • LSA's Semantic Power: LSA can effectively capture the semantic content of free-form verbal interactions, moving beyond simple keyword matching to understand the meaning and relationships between utterances.
  • Automated Tagging is Feasible: While not perfectly matching human agreement, LSA can automatically tag discourse with reasonable accuracy, offering a substantial speedup (hours of manual work reduced to minutes).
  • Discourse Predicts Performance: Specific types of communication (e.g., stating facts, acknowledgments) correlate positively with better team performance, while others (e.g., uncertainty, frequent responses) correlate negatively. This aligns with previous manual analysis findings.
  • Whole-Transcript Analysis is Powerful: Analyzing the semantic similarity of entire team transcripts provides a highly accurate prediction of team performance, suggesting that the overall flow and content of communication are strong indicators of success.

Key Results

  • Automatic Tagging Accuracy: The "LSA+" algorithm (with discourse features) performed within 11-15% of human-human agreement for discourse tagging.
  • Tag-Performance Correlation: Significant correlations were found between tag frequencies and team performance. For example, "Fact" and "Acknowledgement" tags showed positive correlations (0.335 and 0.320, respectively), while "Uncertainty" and "Response" showed negative correlations (-0.321 and -0.460, respectively).
  • Transcript-Based Performance Prediction: The LSA-based prediction of overall team performance correlated strongly with actual performance scores (r = 0.76, p < 0.01). This indicates that LSA can accurately predict how well a team will complete its mission based solely on its communication transcript.

Limitations and Future Work

  • Manual Annotation Limitations: The study acknowledges that the manual annotations had only moderate inter-coder reliability. This difficulty in human agreement also posed challenges for the automated algorithm. Future work includes conducting a more rigorous manual annotation study to improve reliability.
  • Sequential Information: The current approach sometimes loses sequential tag information within turns. Future work aims to incorporate syntax and discourse turn sequences for improved performance.
  • Real-time Prediction: The authors are exploring two promising avenues for real-time performance prediction: integrating speech recognition technology and analyzing inter-turn tag sequences.
  • Broader Applicability: The study suggests that these automated NLP techniques can inform theories of team performance and aid in developing more effective automated team training systems.

Summary updated

BibTeX
@inproceedings{martin2004automated,
  author = {Melanie J. Martin and Peter W. Foltz},
  title = {{Automated Team Discourse Annotation and Performance Prediction Using LSA}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure