Abstract
Researchers from the University of Washington have developed a system to reconstruct high-fidelity 3D models of NBA players from a single broadcast photo. By leveraging a massive new dataset extracted from the NBA2K video game, the system accurately captures complex airborne poses and realistic clothing wrinkles that challenge traditional 3D reconstruction methods. The approach significantly outperforms existing state-of-the-art models in both pose accuracy and mesh fidelity.
Introduction
3D human reconstruction—the process of turning a 2D image into a 3D mesh—has seen massive gains recently, but it still frequently fails in "the wild." Most existing models, such as SMPL, are trained on datasets of people in minimal clothing performing static, staged poses.
Basketball serves as a rigorous "stress test" for these models. Players perform extreme athletic maneuvers like dunks and double-clutch layups, wear loose-fitting jerseys that fold and flap, and frequently occlude their own limbs. Furthermore, most computer vision systems assume subjects are standing on the ground; in basketball, players are often airborne, making it nearly impossible to place them correctly in a 3D scene using standard techniques.
Methods: A Multi-Stage Pipeline
The authors propose a multi-stage pipeline that moves from a 2D crop to a globally positioned 3D mesh. This process integrates pose estimation, physique modeling, and deep skinning to handle the unique challenges of professional sports photography.
- PoseNet: A neural network that predicts 2D and 3D joint locations. Crucially, it also predicts jump status (is the player in the air?) and jump height.
- IdentityNet: This network estimates the player's specific physique. It takes a "template" mesh and deforms it to match the specific player’s height and arm span in the photo.
- SkinningNet (Deep Skinning): Instead of using traditional Linear Blend Skinning, which often looks "rubbery," the authors use a deep learning approach. They train a network to map specific poses to high-resolution geometry, allowing the model to "learn" how a jersey should wrinkle during a jump shot.
- Global Positioning: By detecting court lines and using the predicted jump height, the system solves the "Perspective-n-Point" (PnP) problem to place the 3D player at the correct coordinates on a virtual court.
The NBA2K Dataset
The core contribution of this work is the NBA2K Dataset. Since obtaining 3D scans of active NBA players in mid-air is physically impossible, the researchers used RenderDoc to intercept the graphics instructions of the NBA2K19 video game.
- Scale: 27,144 meshes across 27 different player subjects.
- Diversity: Includes thousands of unique basketball poses (dribbling, shooting, etc.).
- Augmentation: The researchers created 265,765 training examples by blending these game meshes into photos of real basketball courts to help the model generalize to real-world broadcast footage.
High-end sports games are now so graphically sophisticated that their internal assets provide a "silver standard" for training data, offering a ground truth for clothing and pose that doesn't exist in the real world.
Key Results
The method significantly outperforms state-of-the-art (SOTA) models like HMR, SPIN, and PIFu.
- Quantitative: In 3D pose estimation, the method achieved a Mean Per Joint Position Error (MPJPE) of 51.67mm, compared to SPIN’s 88.72mm. Using the Chamfer Distance (a measure of shape matching), the method scored 4.934, while the next best SOTA (SPIN) scored 14.793—a nearly 3x improvement.
- Qualitative: Visual results show that while other models produce "melted" or overly smooth shapes, this method captures distinct fingers, jersey textures, and realistic athletic silhouettes.
Limitations and Future Work
While the system excels at shape reconstruction, it does not yet estimate the player's actual skin texture or facial identity from the photo, currently relying on game textures for visualization. It also struggles with very fine details like hair due to the low resolution of player crops in wide-angle broadcast shots. Future work may involve moving to video-based reconstruction to improve temporal consistency and smooth out jitter in highly occluded scenes.
Summary updated