Multi-Modal Machine Learning for Assessing Gaming Skills in Online Streaming: A Case Study with CS: GO

Longxiang Zhang, Wenping WangarXiv 2023 · DOI · arXiv

Introduction

The rise of online streaming platforms like Twitch has created a massive repository of gameplay data, offering a unique opportunity for talent identification in e-sports. However, the sheer volume of content makes manual skill assessment by scouts impractical. This study explores the use of multi-modal deep learning to automatically classify the skill levels of Counter-Strike: Global Offensive (CS:GO) players using raw video, audio, and chat logs.

Methodology and Architecture

The researchers developed an end-to-end system based on a state-of-the-art lipreading model, adapted for the complexities of gaming data. The architecture integrates multiple data streams to capture the nuances of high-level play.

The multi-modal architecture combining 3D convolutions and ResNets with bidirectional GRUs for video and audio processing.

The model utilizes spatio-temporal CNNs (3D Conv) to extract movement features from video sequences, while ResNet-18 and ResNet-34 backbones process audio and visual information. Bidirectional Gated Recurrent Units (BGRUs) then capture temporal dependencies across both streams. A key technical innovation is the use of KL Divergence Loss to align the audio branch with the more feature-rich video branch, improving overall robustness. Additionally, a Text Prior—implemented via a Bi-LSTM—processes Twitch chat history to provide a baseline skill estimate before visual analysis begins.

Multiview Skill Assessment

Recognizing that different parts of the screen convey different types of information, the authors implemented a "multiview" approach. Instead of analyzing the entire frame, the model focuses on specific UI elements that are indicative of a player's strategic awareness and mechanical skill.

Specific UI regions (minimap, health, kill feed, weapon selection) used in the multiview model to isolate skill-relevant signals.

By cropping regions such as the minimap, health bar, kill feed, and weapon selection (Guns View), the researchers could evaluate which visual cues were most predictive of rank. Their findings indicated that the "Top View" (match scores) and "Guns View" provided the highest precision for skill assessment.

The "Identity Trap" and Key Results

The study utilized a cleaned dataset of 1,620 videos from 268 users, categorized into four ESEA league ranks (A through D). While the best-performing model (LR+KL) achieved an impressive F1-score of 0.850 on a video-based split, a critical flaw emerged during more rigorous testing.

When the model was evaluated on a user-based split—where the test set contained entirely new players not seen during training—the F1-score plummeted to approximately 0.500. This "Identity Trap" suggests that the AI was not primarily learning gameplay skill, but rather identifying specific streamers by their voice, face, or custom overlays. This highlights a significant challenge in personalized AI: distinguishing between a user's identity and their actual performance metrics.

Conclusion and Future Directions

While multi-modal models show promise, current architectures are prone to learning "shortcuts" like user identification. Future work will focus on reframing the task from classification to a relative ranking problem and improving the temporal alignment of chat events with gameplay highlights. Expanding the dataset to include a more diverse range of users is essential for developing models that truly generalize across the e-sports landscape.

Summary updated

BibTeX
@misc{zhang2023multi,
  author = {Longxiang Zhang and Wenping Wang},
  title = {{Multi-Modal Machine Learning for Assessing Gaming Skills in Online Streaming: A Case Study with CS: GO}},
  year = {2023},
  eprint = {2307.12236},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.12236},
}
Figure