Gradient-based 2D-to-3D Conversion for Soccer Videos

Kiana Calagari, Mohamed Elgharib, Piotr Didyk, Alexandre Kaspar, Wojciech Matusik, Mohamed Hefeeda2015 · DOI

TL;DR

This paper introduces a novel, domain-specific method for converting 2D soccer videos into high-quality 3D. By leveraging computer-generated game data for depth information and a gradient-based transfer approach, they achieve significantly better results than general-purpose methods, producing 3D content that is nearly indistinguishable from original 3D footage.

What Problem Does This Paper Address?

The widespread adoption of 3D displays is hampered by a severe lack of 3D content. While manual conversion of 2D footage to 3D is possible, it's prohibitively expensive and time-consuming, especially for live events like sports. Existing automated 2D-to-3D conversion methods often produce low-quality results with noticeable artifacts, making them unsuitable for many viewers. This paper aims to bridge this gap by developing a high-quality, automated conversion system specifically for soccer videos, a domain with unique challenges like complex motion, varied camera angles, and dynamic scenes.

Methods

The core of the proposed method, dubbed "Depth Gradient-based Conversion" (DGC), is a data-driven approach that transfers depth information from a reference database. The key innovation lies in the source of this reference data: instead of relying on difficult-to-obtain real-world 3D video, they utilize high-quality, computer-generated 3D data from sports video games (specifically FIFA13).

Here's a breakdown of their approach:

  • Synthetic Database Creation: They extract 2D frames and corresponding depth maps from FIFA13, creating a large dataset (S-RGBD) that captures diverse soccer scenarios.
  • Block-based Matching: For a given 2D input frame, they search their synthetic database for similar frames. Instead of a global match, they divide the input frame into blocks and find the best matching blocks in the database. This local approach makes the matching more robust to variations in camera angles and scene content.
  • Depth Gradient Transfer: The crucial step is transferring depth gradients (the rate of change in depth) from the matched blocks in the synthetic database to the input frame. This is a departure from methods that transfer absolute depth.
  • Poisson Reconstruction: The transferred depth gradients are then used to reconstruct a dense depth map for the input frame using Poisson reconstruction, a robust technique for solving gradient-based problems.
  • Object Masking and Refinement: To ensure sharp object boundaries and prevent artifacts, they create object masks (identifying players and other key elements) and use them to refine the depth gradients and the Poisson reconstruction process. This helps maintain clear edges and prevents depth blending across objects.
  • Smoothness Constraints: Additional smoothness constraints are applied to the depth map to further refine the output and avoid overly sharp or noisy transitions.

Datasets

  • Synthetic Database (S-RGBD): The paper introduces a novel synthetic dataset created by extracting 2D frames and depth maps from the FIFA13 video game. This dataset contains 16,500 frames and is designed to cover a wide variety of soccer shots and scenarios.
  • Evaluation Datasets:
    • Synthetic Soccer Sequence (Synth): 120 2D+Depth frames extracted from FIFA13, used for objective comparison against ground truth.
  • Real Soccer Sequences: Eight real-world soccer video clips, including long shots, bird's eye views, medium shots, and close-ups.
  • Non-Soccer Sequences: Clips from Tennis, Baseball, American Football, and Field Hockey, used to demonstrate the potential applicability of their method to other sports.

Key Insights and Ideas

  • Domain-Specific Conversion is Key: The paper strongly advocates for domain-specific 2D-to-3D conversion (soccer in this case) over general-purpose methods, arguing it leads to significantly higher quality results.
  • Synthetic Game Data as a Powerful Resource: The most significant insight is the successful use of computer-generated game data as a high-quality, readily available source for depth information. This bypasses the difficulty of acquiring real-world 3D video data for training.
  • Gradient Transfer Outperforms Absolute Depth Transfer: Transferring depth gradients on a block-by-block basis proves more effective than transferring absolute depth values. This allows for finer depth assignment to smaller regions and objects, leading to more natural-looking 3D.
  • Block-based Matching Enhances Robustness: The local, block-based matching strategy makes the system more robust to variations in camera angles, scene content, and the number of objects compared to global matching approaches.
  • Object Masks are Crucial for Sharpness: Explicitly handling object boundaries through masks is essential for preventing artifacts and ensuring clear depth discontinuities, especially after gradient refinement.

Key Results

  • Objective Performance: The DGC method achieves significantly lower Mean Absolute Error (MAE) compared to state-of-the-art methods like Depth Transfer (DT) and its extension (DT+) on both synthetic and real soccer sequences. For instance, on a synthetic sequence, DGC produced much lower MAE than DT and DT+. On real sequences, DGC reduced MAE by up to 19% and 86% compared to DT and DT+, respectively.
  • Subjective Performance: User studies showed that DGC produces 3D videos that are "almost indistinguishable from videos originally shot in stereo." Subjects rated the perceived depth quality and visual comfort as "Excellent" for most soccer sequences.
  • Comparison to Original 3D: When compared to original 3D videos, DGC's conversion was found to be comparable, especially in long shots which constitute a large portion of soccer games. Interestingly, some subjects found DGC more comfortable than original 3D, noting that the "popping out" effect in some original 3D content could be uncomfortable.
  • Outperforms State-of-the-Art: DGC consistently outperformed DT+ in subjective evaluations, with an average advantage of 15 points in medium shots and 12 points in close-ups for depth quality.
  • Computational Efficiency: DGC is significantly faster than DT+, with processing times of 3.53 min/frame for close-ups and 1.86 min/frame for non-close-ups, compared to DT+'s 15.2 min/frame.

Limitations and Future Work

  • Limitations:
    • The current method is domain-specific to soccer. While it shows potential for other sports, a dedicated database for each sport would likely yield better results.
  • Illumination variation can impact field detection and player detection, potentially affecting the accuracy of object masks.
  • Close-up shots are more computationally expensive due to the mask creation step.
  • Future Work:
    • Creating larger and more diverse synthetic databases for different sports.
  • Improving robustness to illumination variations by experimenting with various background subtraction methods.
  • Exploring spatio-temporal multi-resolution schemes to further improve computational complexity, especially for close-up shots.

Summary updated

Abstract

A wide spread adoption of 3D videos and technologies is hindered by the lack of high-quality 3D content. One promising solution to address this problem is to use automated 2D-to-3D conversion. However, current conversion methods, while general, produce low-quality results with artifacts that are not acceptable to many viewers. We address this problem by showing how to construct a high-quality, domain-specific conversion method for soccer videos. We propose a novel, data-driven method that generates stereoscopic frames by transferring depth information from similar frames in a database of 3D stereoscopic videos. Creating a database of 3D stereoscopic videos with accurate depth is, however, very difficult. One of the key findings in this paper is showing that computer generated content in current sports computer games can be used to generate high-quality 3D video reference database for 2D-to-3D conversion methods. Once we retrieve similar 3D video frames, our technique transfers depth gradients to the target frame while respecting object boundaries. It then computes depth maps from the gradients, and generates the output stereoscopic video. We implement our method and validate it by conducting user-studies that evaluate depth perception and visual comfort of the converted 3D videos. We show that our method produces high-quality 3D videos that are almost indistinguishable from videos shot by stereo cameras. In addition, our method significantly outperforms the current state-of-the-art method. For example, up to 20% improvement in the perceived depth is achieved by our method, which translates to improving the mean opinion score from Good to Excellent.
BibTeX
@misc{calagari2015gradient,
  author = {Kiana Calagari and Mohamed Elgharib and Piotr Didyk and Alexandre Kaspar and Wojciech Matusik and Mohamed Hefeeda},
  title = {{Gradient-based 2D-to-3D Conversion for Soccer Videos}},
  year = {2015},
  doi = {10.1145/2733373.2806262},
}
Figure