Temporal Hockey Action Recognition via Pose and Optical Flows

Zixi Cai, Helmut Neher, Kanav Vats, David A. Clausi, John S. Zelek

Abstract

Ice hockey presents unique challenges for automated action recognition due to high-speed movement, bulky equipment, and small player scales in broadcast footage. This paper introduces a novel two-stream deep learning framework that combines human pose estimation with temporal motion data from optical flows. A key innovation is the extension of standard pose models to include the hockey stick as an integral part of the player's skeleton. Evaluated on the new HARPET dataset, the proposed model achieves 85% accuracy in classifying complex maneuvers such as shooting and passing.

Introduction

Action recognition in sports is a critical component of automated analytics, yet ice hockey remains a difficult domain for computer vision. Players move at extreme speeds, leading to significant motion blur, and their bulky uniforms often obscure standard anatomical landmarks. Furthermore, traditional pose estimation models fail to account for the hockey stick, which is the primary tool through which players interact with the game.

To address these issues, this research proposes a specialized architecture that treats the hockey stick as an extension of the human body. By fusing structural pose data with macroscopic motion information from optical flows, the system can distinguish between subtle actions like passing and shooting that are often indistinguishable in static images.

Methodology

The core of the system is a two-stream architecture designed to process short video sequences (three frames) to classify four primary actions: skating forward, skating backward, passing, and shooting.

Overview of the two-stream action recognition framework combining pose estimation and optical flow.

The Pose Stream

The pose stream utilizes a Part Affinity Fields (PAF) model to estimate the player's skeleton. Standard 16-joint human models are insufficient for hockey; therefore, the authors extended the model to 18 joints by adding the "butt end" and the "blade" of the hockey stick. These coordinates, along with the angles between limbs, are transformed into a Latent Joint Feature Vector.

The Optical Flow Stream

To capture the "how" of the movement, the model employs LiteFlowNet to generate optical flow fields between consecutive frames. This provides a dense representation of motion that complements the sparse structural data from the pose stream.

Fusion and Training

Features from both streams are concatenated and processed through fully-connected layers to determine action probabilities. Due to the specialized nature of the task, the authors utilized transfer learning, pre-training the pose model on the MSCOCO dataset before fine-tuning on the newly created HARPET (Hockey Action Recognition Pose Estimation, Temporal) dataset. HARPET consists of approximately 1200 images organized into 400 temporal sequences.

Key Contributions: The Stick as a Limb

A central finding of this work is that localizing the hockey stick is as vital as localizing the player's limbs. By integrating the stick into the pose estimation, the model's accuracy improved by 13%.

The 18-joint pose model integrating the hockey stick as a fundamental part of the player's skeleton.

The inclusion of the stick allows the model to capture the geometric relationship between the player's hands and the tool, which is essential for identifying "shooting" versus "skating." Furthermore, the two-stream approach proved synergistic: adding optical flow to a pose-only model provided a 10% boost in performance, as the flow fields help resolve ambiguities caused by pose estimation errors or motion blur.

Key Results

The integrated model achieved an overall end-to-end accuracy of 85% on the HARPET dataset. Ablation studies confirmed the importance of each component:

  • Full Model: 85% accuracy
  • Without Stick Information: 72% accuracy
  • Without Optical Flow: 75% accuracy

The pose estimation component itself was highly precise, reaching a [email protected] of 87%, validating the effectiveness of transfer learning for domain-specific pose tasks.

Limitations and Future Work

Despite its success, the model has limitations. It currently does not track the puck, which is a critical contextual cue for distinguishing between passes and shots. Future work will focus on expanding the HARPET dataset, incorporating puck tracking, and utilizing longer temporal sequences (e.g., via RNNs) to capture multi-stage actions. Additionally, incorporating environmental context, such as the position of the goal or other players, could further enhance recognition accuracy.

Summary updated

BibTeX
@inproceedings{cai2019temporal,
  author = {Zixi Cai and Helmut Neher and Kanav Vats and David A. Clausi and John S. Zelek},
  title = {{Temporal Hockey Action Recognition via Pose and Optical Flows}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00310},
}
Figure