3M-Game: Multi-Modal Multi-Task Multi-Teacher Learning for Game Event Detection (Student Abstract)

Thye Shan Ng, Feiqi Cao, Soyeon Caren Han

Introduction

Esports titles like League of Legends (LoL) present a significant barrier to entry for new viewers due to their fast-paced nature and complex terminology. While automated game event detection—identifying critical moments like player kills or tower takedowns—could aid accessibility, existing models often struggle. Most rely on single modalities (e.g., video or raw game data) and fail to capture the contextual "vibe" provided by excited commentators or the emotional surge in live chat.

Methodology: The 3M-Game Framework

The authors introduce 3M-Game, a framework utilizing Multi-Modal, Multi-Task, and Multi-Teacher learning. The core innovation lies in a Knowledge Distillation approach that allows a lightweight "student" model to inherit expertise from multiple specialized "teacher" models.

The 3M-Game architecture featuring three specialized teachers and a student model linked by distillation losses.

The framework consists of:

  1. Specialized Teachers: Three independent models are fine-tuned for specific domains: an Audio Teacher (AST) for sound-based action detection, a Chat Teacher (XLM-RoBERTa) for emotion tagging in live streams, and a Transcript Teacher (RoBERTa) for identifying game entities in commentary.
  2. Student Model: A smaller RoBERTa model trained to detect events using only text transcripts.
  3. Knowledge Distillation: During training, the student mimics both the "soft labels" (predictions) and "hidden states" (internal representations) of the teachers. This enables the student to leverage multimodal context—like audio cues and chat sentiment—at inference time without requiring those data streams as direct inputs.

Experimental Setup

The model was evaluated using the Game-MUG dataset, comprising League of Legends matches from Twitch and YouTube. Raw audio was converted to text using OpenAI’s Whisper. The final dataset included 28,145 instances, split 95/5 for training and testing.

Key Results and Discussion

The 3M-Game framework demonstrates a substantial performance leap over the baseline (Zhang et al. 2024).

Performance comparison of 3M-Game against baseline and ablation variants.

As shown in the results table, the model achieved an overall precision of 0.495. Most notably, when focusing on "Main Events" (Kills, Towers, Dragons), the precision reached 0.638, more than tripling the baseline's 0.185.

Ablation studies highlight the importance of each modality. Removing the Chat teacher caused the most significant performance drop (from 0.495 to 0.325), suggesting that viewer sentiment is a critical indicator of game events. PCA visualizations further confirmed that the distillation process successfully organized the model's internal embeddings into distinct clusters corresponding to different event types.

Conclusion and Future Work

3M-Game proves that multi-teacher distillation can effectively "bake" multimodal knowledge into a text-only student model, significantly improving event detection in complex esports environments. Future research will focus on reducing latency for real-time applications and testing the framework's generalizability across different game genres like First-Person Shooters.

Summary updated

Abstract

Esports has rapidly emerged as a global phenomenon with an ever-expanding audience on livestream platforms. However, due to the complex nature of the game, it becomes challenging for newcomers to comprehend the gaming situation. This research introduces a 3M-Game that integrates multi-modal (MM) information from the livestream platform, including chat and livestream, to uncover the event. While conventional MM models typically prioritise aligning MM data through concurrent training towards a unified objective, our framework leverages multiple independent teachers trained on different tasks to accomplish game event detection. The results show the effectiveness of the proposed framework. The code and appendix are in https://github.com/adlnlp/3m_game.
BibTeX
@inproceedings{ng2025m,
  author = {Thye Shan Ng and Feiqi Cao and Soyeon Caren Han},
  title = {{3M-Game: Multi-Modal Multi-Task Multi-Teacher Learning for Game Event Detection (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35283},
}
Figure