Volley3D: Unified 3D Full-Sequence Reconstruction from Monocular Broadcasts

Noâm Boussouf, Adrien Maglo, Mohamed Chaouch, Hideo SaitoarXiv 2026 · arXiv

TL;DR

Volley3D is a modular, physics-grounded pipeline that reconstructs full 3D volleyball scenes—including player poses and ball trajectories—from a single uncalibrated broadcast video. By explicitly modeling jump dynamics and anchoring ball flight to player hand vertices, it overcomes the severe depth ambiguities and perspective drift that plague standard monocular reconstruction methods.

What Problem Does This Paper Address?

Reconstructing sports in 3D from a single camera is a "notoriously ill-posed" problem because depth information is lost during projection. While this has been explored for "ground-bound" sports like soccer or tennis, volleyball presents unique challenges:

  1. High-Altitude Jumps: Standard methods assume players stay on the ground plane. When a volleyball player jumps, a standard camera projection misinterprets their vertical rise as a retreat in depth, causing massive "perspective drift."
  2. Complex Ball Dynamics: Volleyball features unpredictable multi-contact sequences (sets, spikes, blocks) and fast-moving projectiles.
  3. Data Scarcity: There are no large-scale datasets providing 3D ground truth for both players and balls in volleyball, making end-to-end deep learning unfeasible.

Methods

The authors propose a two-stage framework that replaces massive data requirements with physical laws.

Stage 1: Player Reconstruction & Jump Handling The system first tracks players in 2D and estimates their 3D poses in camera space. To fix the depth issue, they introduce a physics-informed jump handling module. It uses a pose-based event spotter to identify takeoff and landing frames. During the jump, the system overrides faulty ground-plane projections by interpolating the player's 3D trajectory using Newtonian equations of motion (parabolic arcs).

Stage 2: Context-Aware Ball Trajectory Estimation Instead of treating the ball independently, Volley3D anchors it to the players.

  • Segmentation: A Temporal Convolutional Network (TCN) breaks the global ball track into "sub-trajectories" separated by hits or bounces.
  • Collision Matching: A heuristic algorithm identifies which player and which specific hand (or the court surface) caused the deflection.
  • Aerodynamic Solver: The ball's 3D path is solved as a Boundary Value Problem (BVP). By using the reconstructed 3D hand positions as start and end points, the system "lifts" the 2D ball detections into a spatiotemporally coherent 3D scene without needing perfect camera calibration.
Figure 2. Overview of our proposed two-stage monocular 3D reconstruction pipeline. In the first stage, we extract and position the athletes in 3D world-space, combining smoothed homographic projections with a pose-based jump spotting module to handle aerial phases. In the second stage, we segment 2D ball tracks and explicitly anchor them to the reconstructed athletes, using the predicted 3D collision points to physically constrain the final 3D ballistic trajectory. The blue modules highlight the contributions introduced in this paper.

Datasets

  • Volleyball Dataset (Saad et al.): Substantially extended with manual annotations for court keypoints (1,500 frames), jump phases (50 sequences), and over 1,600 ball deflection events.
  • VolliQ: A semi-professional multi-view dataset used for 3D validation. The authors manually annotated 12,000 ball bounding boxes across 3,000 multi-view frames to establish a real-world 3D benchmark.
  • SportsMOT: Used to increase visual diversity for training the tracking modules.

Key Insights and Ideas

  • Jumps as Projective Failures: Figure 3 (below) perfectly illustrates the "aha moment": raw homography (grey) shows a player "drifting" back and forth during a jump. By applying Newtonian physics (red), the system recovers the true vertical motion.
  • Players as 3D Anchors: The most significant insight is that reconstructed 3D players can serve as high-quality "boundary conditions" for the ball. Anchoring the ball to a player's hand is far more robust than trying to minimize 2D reprojection errors from a noisy, moving camera.
  • Physics-Based Lifting: In the absence of 3D training data, hard-coding aerodynamic drag and gravity into the optimization allows for high-fidelity reconstruction that purely data-driven methods currently cannot match in this domain.
The effect of physics-informed jump correction. Raw homography (grey) shows significant depth drift during airborne phases (yellow), which the proposed method (red) successfully corrects.

Key Results

  • Spatial Accuracy: The jump correction module reduced Mean Absolute Error (MAE) in court depth from 1.05m (standard homography) to just 0.23m.
  • Ball Tracking: On the VolliQ benchmark, Volley3D achieved a 3D RMSE of 0.27m. For comparison, the state-of-the-art monocular baseline (MonoTrack) suffered from a massive 1.56m error due to depth ambiguity.
  • Immersive Replays: The system successfully enables realistic novel-view synthesis, allowing a single broadcast feed to be viewed from any angle in a VR/AR environment.
Figure 1. Monocular 3D collective reconstruction from broadcast volleyball video. From a single uncalibrated broadcast sequence (a), our framework recovers 3D player poses, jump dynamics, and physical ball trajectories in global court coordinates, enabling realistic novel-view synthesis from multiple vantage points (b, c).

Limitations and Future Work

  • Net Interactions: The current model does not yet account for ball-net contacts, which are frequent in professional play.
  • Real-Time Processing: The pipeline is currently designed for offline analysis. The authors aim to optimize it for real-time AR/VR broadcasting applications.
  • End-to-End Evaluation: Future work includes testing on a larger dataset that pairs broadcast camera views with synchronized 3D ground truth for more comprehensive validation.

Summary updated

Abstract

3D reconstruction from single-view sports broadcast is a pivotal capability for automated analytics and immersive AR/VR replays, yet notoriously ill-posed due to difficult-to-calibrate moving cameras and depth ambiguity. Unlike widely studied racket sports, team sports such as volleyball feature unpredictable multi-contact sequences and high altitude jumps. These dynamics categorically invalidate the standard planar projection assumptions relied upon by existing methods. To resolve these limitations without 3D training supervision, we propose a complete, physics-grounded pipeline tailored to monocular broadcast volleyball. First, we include a correction module to the perspective drift induced by ground-plane projection by coupling temporal jump spotting with Newtonian interpolation for 3D player placement. Second, for 3D ball trajectory estimation, we design a dedicated sub-trajectory segmentation architecture alongside a collision-matching algorithm. By anchoring physical equations to reconstructed 3D hand vertices, we lift noisy ball 2D detections into a spatiotemporally coherent 3D scene. Evaluations on multi-camera benchmarks demonstrate our approach significantly reduces 3D spatial drift compared to 2D-reprojection baselines.
BibTeX
@misc{boussouf2026volley,
  author = {Noâm Boussouf and Adrien Maglo and Mohamed Chaouch and Hideo Saito},
  title = {{Volley3D: Unified 3D Full-Sequence Reconstruction from Monocular Broadcasts}},
  year = {2026},
  eprint = {2610.38358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38358},
}
Figure