Introduction
The rise of online streaming platforms like Twitch has created a massive repository of gameplay data, offering a unique opportunity for talent identification in e-sports. However, the sheer volume of content makes manual skill assessment by scouts impractical. This study explores the use of multi-modal deep learning to automatically classify the skill levels of Counter-Strike: Global Offensive (CS:GO) players using raw video, audio, and chat logs.
Methodology and Architecture
The researchers developed an end-to-end system based on a state-of-the-art lipreading model, adapted for the complexities of gaming data. The architecture integrates multiple data streams to capture the nuances of high-level play.
The model utilizes spatio-temporal CNNs (3D Conv) to extract movement features from video sequences, while ResNet-18 and ResNet-34 backbones process audio and visual information. Bidirectional Gated Recurrent Units (BGRUs) then capture temporal dependencies across both streams. A key technical innovation is the use of KL Divergence Loss to align the audio branch with the more feature-rich video branch, improving overall robustness. Additionally, a Text Prior—implemented via a Bi-LSTM—processes Twitch chat history to provide a baseline skill estimate before visual analysis begins.
Multiview Skill Assessment
Recognizing that different parts of the screen convey different types of information, the authors implemented a "multiview" approach. Instead of analyzing the entire frame, the model focuses on specific UI elements that are indicative of a player's strategic awareness and mechanical skill.
By cropping regions such as the minimap, health bar, kill feed, and weapon selection (Guns View), the researchers could evaluate which visual cues were most predictive of rank. Their findings indicated that the "Top View" (match scores) and "Guns View" provided the highest precision for skill assessment.
The "Identity Trap" and Key Results
The study utilized a cleaned dataset of 1,620 videos from 268 users, categorized into four ESEA league ranks (A through D). While the best-performing model (LR+KL) achieved an impressive F1-score of 0.850 on a video-based split, a critical flaw emerged during more rigorous testing.
When the model was evaluated on a user-based split—where the test set contained entirely new players not seen during training—the F1-score plummeted to approximately 0.500. This "Identity Trap" suggests that the AI was not primarily learning gameplay skill, but rather identifying specific streamers by their voice, face, or custom overlays. This highlights a significant challenge in personalized AI: distinguishing between a user's identity and their actual performance metrics.
Conclusion and Future Directions
While multi-modal models show promise, current architectures are prone to learning "shortcuts" like user identification. Future work will focus on reframing the task from classification to a relative ranking problem and improving the temporal alignment of chat events with gameplay highlights. Expanding the dataset to include a more diverse range of users is essential for developing models that truly generalize across the e-sports landscape.
Summary updated