Part-Based Player Identification Using Deep Convolutional Representation and Multi-Scale Pooling

Arda Senocak, Tae-Hyun Oh, Jun-Sik Kim, In So Kweon

TL;DR

This paper tackles the challenging problem of automatically identifying basketball players in broadcast videos, where visual cues like faces and jersey numbers are often obscured. They propose a novel approach that combines deep convolutional features extracted at multiple scales with a part-based representation, achieving state-of-the-art results of up to 96% accuracy.

What Problem Does This Paper Address?

Identifying individual players in broadcast sports videos is surprisingly difficult. Unlike close-up camera views where faces and jersey numbers are clear, broadcast footage often presents players from a distance, wearing identical uniforms. This makes it hard to distinguish them based on traditional cues. Furthermore, players' appearances change rapidly due to their actions and varying body poses. Existing methods often rely on manual labeling or struggle with occlusions and the lack of clear identifying features, necessitating a more robust and automated approach.

Methods

The core of the proposed system lies in a novel network architecture that leverages deep learning for player representation. Here's a breakdown:

  • Multi-Scale Deep Convolutional Representations: Instead of processing a player image at a single scale, the authors extract features from the player at multiple resolutions. This allows the system to capture both coarse and fine-grained details. They use a modified AlexNet architecture to extract activations from seven different scales.
  • Fisher Vector Encoding: To effectively encode these multi-scale activations into a discriminative representation, they employ the Fisher Vector method. This technique is known for its ability to capture rich statistical information from local features and has been shown to be effective in image classification tasks.
  • Part-Based Representation: Through visualization, the authors observed that distinctive player features often lie in specific body parts not covered by the jersey (e.g., head, legs, feet). To exploit this, they use Convolutional Pose Machines (CPM) to detect key body parts. Features from these detected parts are then extracted and encoded using the same Fisher Vector approach.
  • Late Fusion: Finally, the holistic body representation (fbody) and the part-based representation (fparts) are combined through concatenation to form the final player representation (fplayer). This fusion allows the system to leverage both global and local discriminative information.

Datasets

The authors introduce a new dataset for training and evaluating their player identification system. This dataset was collected from a commercial broadcast of a 2014-2015 NBA regular season game between the Denver Nuggets and the Houston Rockets. They specifically focused on sequences captured from the broadcast camera view. Human annotators were tasked with drawing bounding boxes around players and labeling them in sampled consecutive frames. For training, they sampled 2500 instances per player from the first half of the game, and for testing, they used clips of varying lengths from the second half.

Key Insights and Ideas

  • Body parts matter: Visualizing the learned features revealed that specific, uncovered body parts (like heads and legs) are crucial for distinguishing players, even when uniforms are identical. This insight directly motivated the part-based representation.
  • Multi-scale is key: Capturing features at various resolutions is essential for discerning the subtle differences between players in broadcast footage.
  • Fisher Vectors are powerful: The Fisher Vector encoding effectively compresses rich, multi-scale convolutional features into a compact and discriminative representation.
  • Late fusion enhances performance: Combining holistic body features with part-specific features leads to a more robust and accurate player identification system.

Key Results

The proposed method achieves impressive results:

  • State-of-the-art accuracy: The combined body and part-based representation achieves up to 96% accuracy on NBA basketball clips.
  • Superior to baseline methods: Their approach significantly outperforms traditional methods like Bag-of-Words (BoW) with DSIFT features and even single-scale deep learning features from architectures like AlexNet, ResNet-50, and VGG-16.
  • Impact of multi-scale: Using 7 scales for feature extraction yields the best performance, with marginal gains beyond that.
  • Qualitative success: The system demonstrates accurate identification in challenging scenarios, as shown in their example videos, correctly identifying players even with occlusions and dynamic poses.

Limitations and Future Work

  • No temporal information: The current system identifies players on a per-frame basis, ignoring temporal consistency. The authors suggest that incorporating temporal cues or using assignment problems could further improve accuracy.
  • Potential for RNNs: They propose that using Recurrent Neural Networks (RNNs) to process consecutive frames could also enhance performance.
  • Dataset limitations: While they introduced a new dataset, it's derived from a single commercial broadcast. Expanding to more diverse datasets and sports could be beneficial.
  • Player-specific insights: The paper notes that one player (Motiejunas) consistently had higher accuracy, potentially due to his Caucasian appearance distinguishing him from other players on his team. This highlights how specific visual cues can influence performance.

Summary updated

Abstract

This paper addresses the problem of automatic player identification in broadcast sports videos filmed with a single side-view medium distance camera. Player identification in this setting is a challenging task because visual cues such as faces and jersey numbers are not clearly visible. Thus, this task requires sophisticated approaches to capture distinctive features from players to distinguish them. To this end, we use Convolutional Neural Networks (CNN) features extracted at multiple scales and encode them with an advanced pooling, called Fisher vector. We leverage it for exploring representations that have sufficient discriminatory power and ability to magnify subtle differences. We also analyze the distinguishing parts of the players and present a part based pooling approach to use these distinctive feature points. The resulting player representation is able to identify players even in difficult scenes. It achieves state-of-the-art results up to 96% on NBA basketball clips.
BibTeX
@inproceedings{senocak2018part,
  author = {Arda Senocak and Tae-Hyun Oh and Jun-Sik Kim and In So Kweon},
  title = {{Part-Based Player Identification Using Deep Convolutional Representation and Multi-Scale Pooling}},
  booktitle = {Workshops @ CVPR},
  year = {2018},
  url = {https://openreview.net/forum?id=B1-Qmkz_ZB},
}
Figure