TL;DR
This paper introduces a novel "3M" framework for detecting game events in esports livestreams by integrating multiple modalities (audio, chat, commentary). Instead of a single model trying to learn everything, 3M uses specialized "teacher" models for each modality and then distills their knowledge into a single "student" model, leading to improved comprehension of complex game situations.
What Problem Does This Paper Address?
Esports viewership is booming, but for newcomers, understanding the fast-paced action, chaotic chat, and rapid commentary can be overwhelming. Existing approaches often focus on single modalities or try to align all data into one model, which can be suboptimal. This paper tackles the challenge of creating a comprehensive understanding of game events by effectively combining information from audio, player chat, and commentator transcripts, aiming to make esports more accessible.
Methods
The core of the proposed solution is the "3M" (Multi-modal Multi-task Multi-teacher) framework. It works by:
- Data Segmentation: Livestream data (audio, chat, commentary) is first segmented into discrete chunks. OpenAI's Whisper model is used to transcribe audio into text, which then helps define these chunks.
- Multi-Teacher Setup: Three specialized "teacher" models are trained independently:
- Audio-based Action Detection Teacher: Uses an Audio Spectrogram Transformer (AST) to identify events from game audio.
- Chat-based Emotion Tagging Teacher: Uses XLM-ROBERTa to analyze chat for emotions and emotes.
- Transcript-based Game Entity Tagging Teacher: Uses RoBERTa to identify game-specific terminology in the commentary transcripts.
- Knowledge Distillation: The expertise from these three teachers is then transferred to a single "student" model. This is done using two distillation losses:
- Multi-teacher Hidden Loss: Aligns the hidden states of the teacher models with the student model's layers.
- Multi-teacher Distillation Loss: Distills the "soft labels" (predictions) from the teachers to the student.
- Task-Specific Loss: A final loss function combines the distillation losses with a task-specific loss that trains the student model directly for game event detection using the transcripts.
This multi-teacher approach is novel because it leverages specialized knowledge from independently trained models, contrasting with conventional methods that often use a single, unified training objective for all modalities.
Datasets
The study uses a dataset curated from League of Legends (LoL) matches broadcast on Twitch and YouTube. This dataset includes game match event logs, audio data, and textual discussions, spanning 45 distinct esports tournaments and 216 unique LoL matches. The data is segmented into 28,145 instances, split into a 95/5 train/test set. The paper also references the Game-MUG dataset for comparison.
Key Insights and Ideas
- Specialization is Key: The paper strongly suggests that using multiple, specialized "teacher" models for different modalities (audio, chat, text) and tasks is more effective than a single monolithic model trying to learn everything.
- Distillation for Efficiency: Knowledge distillation allows the complex knowledge from multiple teachers to be compressed into a single, more manageable student model, making the system more efficient.
- Handling "OTHER" Events: The "OTHER" category, representing periods with no significant actions, is crucial. The proposed 3M framework handles this category well, whereas a baseline model struggles significantly when it's included.
- Modality Synergy: While individual teacher combinations might excel at specific event types (e.g., Chat+Transcript for "OTHER"), the full 3M framework with all modalities, despite potentially lower per-label precision, achieves a higher overall average precision due to better discrimination.
Key Results
- The 3M framework demonstrates a "notably superior performance" compared to the baseline Game-MUG model, especially when the "OTHER" event category is included.
- When the "OTHER" label is removed from the dataset, performance across all events improves for the 3M framework. For example, the precision score for "TOWER" events increased from 0.220 to 0.390.
- An ablation study (Table 3) shows that different combinations of teachers impact performance differently for specific event types. For instance, Audio+Chat excels for "TOWER," while Chat+Transcript is better for "OTHER."
- The paper notes that "KILL" events are detected with higher accuracy, likely due to more audience and commentator attention, while "TOWER" and "DRAGON" events remain challenging.
Limitations and Future Work
- "TOWER" and "DRAGON" Detection: The paper acknowledges that the classification accuracy for "TOWER" and "DRAGON" events remains relatively low. This is hypothesized to be because these events may not provoke as much immediate reaction or have as many discernible features in the data compared to "KILL" events.
- Suboptimal Performance: The authors state that the suboptimal performance for certain labels is a "pertinent avenue for further research."
- Noise Introduction: The addition of teachers, while enhancing overall performance, might introduce noise in separate modalities, leading to lower performance for specific labels. This suggests a need for more sophisticated ways to manage teacher contributions.
Summary updated