CaddieSet: A Golf Swing Dataset with Human Joint Features and Ball Information

Seunghyeon Jung, Seoyoung Hong, Jiwoo Jeong, Seungwon Jeong, Jaerim Choi, Hoki Kim, Woojin Lee

Abstract

CaddieSet is a novel dataset that connects golfer kinematics with ball trajectory data. By pairing video-extracted joint features across eight swing phases with precise launch monitor metrics, the researchers provide a framework for explainable AI in golf instruction. This approach moves beyond simple posture comparison to provide actionable, quantitative feedback on how specific body movements influence ball flight.

Introduction

In golf instruction, "the ball trajectory doesn't lie." However, AI-based analysis has historically struggled to link mechanical causes to flight outcomes. Existing research often relies on cumbersome sensors or simple posture comparisons against professionals, which fail to explain why a specific posture leads to a particular shot. The primary bottleneck has been the lack of paired datasets linking detailed human kinematics to resulting ball flight parameters like speed, spin, and angle.

Methods and Pipeline

The researchers developed a multi-stage pipeline to transform raw video into interpretable insights. The process begins with Swing Sequencing using a fine-tuned SwingNet to segment videos into eight critical phases (e.g., Address, Impact, Finish). Subsequently, Pose Estimation via Faster R-CNN and HRNet extracts 17 joint coordinates.

The CaddieSet pipeline: from raw video input to explainable feedback based on joint features and ball information.

Feature Engineering

To ensure interpretability, the team worked with golf experts to define 15 key metrics, such as Shoulder Angle and Hip Rotation. These metrics are calculated as scale-invariant angles or ratios, allowing for consistent analysis across different body types.

Visualization of the 15 key swing metrics across different phases of the golf swing.

Explainable Modeling

The core of the analysis uses Neural Additive Models (NAMs). Unlike "black box" deep learning models, NAMs treat the influence of each joint feature as a sum of independent functions. This allows the system to quantify exactly how much a specific movement, like head displacement, contributes to a shot's outcome.

The CaddieSet Dataset

CaddieSet comprises 924 shots from eight individuals using various clubs (drivers, woods, and irons). Unlike previous datasets like GolfDB, CaddieSet includes ground truth data from camera-based launch monitors, capturing Ball Speed, Direction Angle, Spin Axis, and Carry Distance.

Key Results and Insights

The study yielded several significant findings:

  • Feature-Based Superiority: Models using the 15 domain-specific metrics often outperformed image-based models like Vision Transformers (ViT). For instance, a Random Forest model achieved a Mean Squared Error (MSE) of 8.80 for Ball Speed, compared to 48.77 for ViT.
  • Optimal Range Discovery: NAMs allowed researchers to identify "optimal zones" for joint metrics. They found a Stance Ratio of ~1.32 minimizes directional deviation, while ~1.85 maximizes speed.
  • Actionable Feedback: The model provides specific instructions (e.g., "Adjust head position by X amount") rather than vague heatmaps, leading to measurable improvements in shot consistency during real-world validation.

Limitations and Future Work

While promising, the dataset is currently limited to eight golfers. Future work aims to increase diversity in body types and skill levels. Additionally, the researchers plan to refine the feedback loop for real-time coaching and explore performance in varying outdoor environmental conditions.

Summary updated

BibTeX
@inproceedings{jung2025caddieset,
  author = {Seunghyeon Jung and Seoyoung Hong and Jiwoo Jeong and Seungwon Jeong and Jaerim Choi and Hoki Kim and Woojin Lee},
  title = {{CaddieSet: A Golf Swing Dataset with Human Joint Features and Ball Information}},
  booktitle = {Workshops @ CVPR},
  year = {2025},
}
Figure