Early Detection of Injuries in MLB Pitchers from Video

A. J. Piergiovanni, Michael S. Ryoo

Abstract

Injuries are a major cost in sports. Teams spend millions of dollars every year on players who are hurt and unable to play, resulting in lost games, decreased fan interest and additional wages for replacement players. Modern convolutional neural networks have been successfully applied to many video recognition tasks. In this paper, we introduce the problem of injury detection/prediction in MLB pitchers and experimentally evaluate the ability of such convolutional models to detect and predict injuries in pitches only from video data. We conduct experiments on a large dataset of TV broadcast MLB videos of 20 different pitchers who were injured during the 2017 season. We experimentally evaluate the model's performance on each individual pitcher, how well it generalizes to new pitchers, how it performs for various injuries, and how early it can predict or detect an injury.

Introduction: The High Cost of Pitcher Injuries

Injuries in Major League Baseball (MLB) represent a half-billion-dollar annual problem, with pitcher-related ailments accounting for nearly half of those costs. While biomechanical sensors and medical imaging provide high-fidelity data for injury detection, they are invasive and prohibited during live competition. Consequently, there is a significant need for non-invasive methods that can monitor player health using existing infrastructure, such as broadcast video.

The primary challenge in using broadcast video is the presence of "noise" and dataset bias. A standard neural network might inadvertently "cheat" by associating injury status with irrelevant cues like the scoreboard (pitch count), team uniforms, or specific stadium backgrounds. To address this, researchers from Indiana University developed a deep learning approach designed to isolate a pitcher's biomechanics from these confounding variables.

Scoreboards in broadcast video provide "cheating" cues like pitch counts that models must learn to ignore.

Methods: Isolating Motion from Appearance

The researchers utilized a 3D spatio-temporal Convolutional Neural Network (I3D), a state-of-the-art architecture for video recognition. The core innovation lies in a rigorous preprocessing pipeline aimed at neutralizing dataset bias and forcing the model to focus strictly on motion patterns:

  1. Spatial Cropping: Videos are tightly cropped around the pitcher to eliminate stadium backgrounds and scoreboards.
  2. Grayscale Conversion: Color information is removed to prevent the model from overfitting to specific team colors or uniforms.
  3. Optical Flow: Instead of raw RGB frames, the model processes "optical flow" maps. These maps represent the movement of pixels between consecutive frames, ensuring the network learns from dynamic biomechanical signatures rather than static appearance.
The preprocessing pipeline converts raw video into optical flow maps to isolate motion patterns from appearance bias.

The task was framed as a binary classification: identifying whether a pitch was "healthy" or "injured." "Injured" pitches were defined as the final $k$ pitches thrown before a pitcher was officially placed on the disabled list.

Experimental Setup and Dataset

The study utilized a custom dataset from the 2017 MLB season, comprising 5,479 pitches from 20 different pitchers (12 left-handed, 8 right-handed). The dataset covered 10 distinct injury types, ranging from UCL tears to hamstring strains. The model was pre-trained on the Kinetics dataset—a large-scale video action dataset—before being fine-tuned on the MLB-specific data.

Key Insights and Results

The experiments yielded several critical findings regarding the feasibility of video-based injury detection:

  • Neutralizing Bias: The authors demonstrated that models trained on raw RGB video were highly biased, often predicting game identity rather than injury status. The combination of cropping and optical flow successfully mitigated this "appearance trap."
  • Generalization and Handedness: While models trained on right-handed pitchers initially failed on left-handed pitchers, the researchers found that horizontally flipping the video allowed for effective cross-handed generalization.
  • Injury Sensitivity: The model was most effective at detecting injuries that significantly alter whole-body mechanics, such as hamstring and back strains. Conversely, it struggled with localized issues like finger blisters, which have minimal impact on visible throwing motion.
  • Performance Metrics: For individual pitchers, the model achieved high accuracy, with F1 scores reaching 0.86 for Clayton Kershaw and 0.97 for Boone Logan. When tested on entirely unseen pitchers, the model maintained a respectable F1 score of approximately 0.71.
  • The Prediction Window: The "injury signal" in mechanics was most detectable within the 10–30 pitches immediately preceding an injury. Beyond a 50-pitch window, the mechanical differences were negligible, suggesting that breakdown occurs relatively close to the point of clinical failure.

Conclusion and Future Directions

This research demonstrates that 3D CNNs can detect subtle biomechanical "tells" indicative of impending injury from standard broadcast video. While the current dataset is relatively small, the results suggest that scaling the approach to more pitchers could further improve generalization. Future work will focus on making the model's outputs more interpretable for coaching staffs—moving from a simple "injured" flag to specific biomechanical feedback—and exploring sequential analysis to capture the cumulative effects of fatigue over multiple innings.

Summary updated

Abstract

Injuries are a major cost in sports. Teams spend millions of dollars every year on players who are hurt and unable to play, resulting in lost games, decreased fan interest and additional wages for replacement players. Modern convolutional neural networks have been successfully applied to many video recognition tasks. In this paper, we introduce the problem of injury detection/prediction in MLB pitchers and experimentally evaluate the ability of such convolutional models to detect and predict injuries in pitches only from video data. We conduct experiments on a large dataset of TV broadcast MLB videos of 20 different pitchers who were injured during the 2017 season. We experimentally evaluate the model's performance on each individual pitcher, how well it generalizes to new pitchers, how it performs for various injuries, and how early it can predict or detect an injury.
BibTeX
@inproceedings{piergiovanni2019early,
  author = {A. J. Piergiovanni and Michael S. Ryoo},
  title = {{Early Detection of Injuries in MLB Pitchers from Video}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=_MEb0jendB},
}
Figure