TL;DR
This paper introduces TVCalib, a novel method for camera calibration in soccer broadcasts that directly estimates camera pose and focal length from sports field segments, bypassing the need for explicit keypoint correspondences. This approach achieves state-of-the-art results in sports field registration, enabling richer applications like augmented reality and sports analytics.
What Problem Does This Paper Address?
Accurately registering a 3D sports field onto 2D broadcast video frames is crucial for many applications, from sports analytics to augmented reality. Traditional approaches often rely on estimating a homography, which maps a planar object to an image. However, this is insufficient for estimating full camera parameters (pose and focal length), especially in broadcast scenarios where cameras are dynamic and focal lengths change. Existing methods that attempt camera calibration often require extensive training data or rely on refining initial homography estimates, which can be suboptimal. The challenge lies in accurately estimating camera parameters from a single, potentially uncalibrated, broadcast frame using the sports field itself as the calibration object, without relying on explicit 3D-2D keypoint correspondences.
Methods
The core of TVCalib is a differentiable objective function that minimizes the segment reprojection error. Instead of using discrete keypoints, it leverages labeled segments (lines, points, and point clouds) of the soccer field. The process involves:
- Segment Localization: Using a DeepLabV3 ResNet model to perform instance segmentation of visible field segments.
- Point Selection: Selecting representative points from these segments (e.g., two points for lines, multiple for point clouds) to form correspondences.
- Calibration Module: A novel module that iteratively optimizes camera parameters (Field of View, translation, pan, tilt, roll) by minimizing the Euclidean distance between annotated segment points and their reprojected counterparts in the image. This is achieved through gradient descent.
- Self-Verification: A mechanism to automatically reject images where the optimization fails to converge to a satisfactory solution, controlled by a threshold on the reprojection loss.
- Multiple Initialization: To handle various camera positions (center, left, right), the method can start optimization from multiple initial camera parameter sets and select the best result.
A key innovation is the "keypoint-less" approach, which directly uses segment information and avoids the difficulties of finding precise keypoints in broadcast footage.
Datasets
The paper evaluates TVCalib on two established soccer datasets:
- SN-Calib dataset: A large dataset derived from SoccerNet videos, offering diverse camera locations and annotated field segments.
- WC14 dataset: A benchmark dataset containing broadcast FIFA World Cup 2014 videos with manually annotated homography matrices.
The authors also mention using ground-truth masks from SN-Calib annotations for some baseline comparisons.
Key Insights and Ideas
- Segments over Keypoints: Directly using segments (lines, points) for calibration is more robust than relying on discrete keypoints, especially in challenging broadcast scenarios.
- Differentiable Reprojection Loss: A differentiable loss function that measures the distance between reprojected segments and their image annotations allows for end-to-end optimization of camera parameters.
- One-Step Calibration: TVCalib performs camera calibration in a single step, avoiding the iterative refinement often seen in homography-based methods.
- Self-Verification for Robustness: The self-verification step is crucial for filtering out images where calibration is unreliable, leading to more robust results.
- Multiple Initializations Improve Performance: Starting the optimization from multiple camera positions significantly improves calibration accuracy compared to a single initialization.
Key Results
TVCalib demonstrates superior performance compared to state-of-the-art methods on both SN-Calib and WC14 datasets.
- Camera Calibration: Achieved significantly higher Accuracy@threshold (AC@t) scores for camera calibration, outperforming baselines like Chen and Little [6] and Jiang et al. [23] by several percentage points across different thresholds (e.g., AC@5 of 62.7% vs. 50.6% on SN-Calib-test-center with GT segmentation).
- Homography Estimation: Also showed competitive results in homography estimation, though the paper emphasizes that camera calibration offers richer information.
- Impact of Segment Localization: While using predicted segments slightly reduces accuracy compared to ground truth, the performance drop is manageable, indicating the effectiveness of the segment localization module.
Limitations and Future Work
The authors acknowledge several limitations:
- Outlier Sensitivity: The current approach relies on accurate segment localization and does not explicitly handle outliers in the segmentation.
- Limited Camera Initialization: The investigation of camera initialization was primarily focused on cameras located on the main tribune.
- Lens Distortion: While the framework can accommodate lens distortion, joint learning of distortion parameters was not deeply investigated.
- Rejection Rate: For a small fraction of images, the method rejects samples due to local minima or other optimization issues.
Future work includes:
- Investigating temporal consistency for improved robustness and speed.
- Applying the method to other sports.
- Integrating TVCalib into end-to-end deep learning frameworks.
- Further exploring lens distortion modeling.
Summary updated