Live RGB-D camera tracking for television production studios

Tommi Tykkälä, Andrew I. Comport, Joni-Kristian Kämäräinen, Hannu HartikainenJVCIR 2014 · DOI · OpenReview

TL;DR

This paper presents an affordable, real-time camera tracking system for live TV production studios using RGB-D sensors. It tackles the high cost and complexity of traditional matchmoving by introducing a keyframe-based approach that significantly reduces drift and improves robustness, even in challenging studio environments.

What Problem Does This Paper Address?

Traditional matchmoving, a technique used to integrate virtual elements into live broadcasts, is typically expensive and time-consuming, often requiring specialized hardware and manual post-processing. While existing online solutions exist, they can cost hundreds of thousands of dollars. The goal of this research is to democratize this technology by creating an affordable, real-time system that uses readily available RGB-D sensors and low-end GPUs, making advanced virtual production techniques accessible to a wider range of studios.

Methods

The core of the proposed system is a keyframe-based RGB-D tracking method. Instead of relying solely on incremental tracking, which can accumulate drift over time, the system first builds a dense keyframe model of the static studio scene. This model is created using image-based dense tracking and bundle adjustment. For real-time tracking, the system registers the current RGB-D measurement against the nearest keyframe in the pre-built database. This approach effectively eliminates time-evolving global drift.

Key technical contributions include:

  • Keyframe-based tracking: A static studio scene is modeled as a database of RGB-D keyframes. Real-time pose estimation is then performed by registering the current frame to the nearest keyframe, preventing drift.
  • Dense RGB-D processing for real-time performance: The algorithms are optimized for low-end GPUs, enabling real-time operation at 30Hz. This involves efficient parallelization of computations.
  • Robustness to dynamic elements: The system uses segmentation-based weights derived from depth and intensity differences to reject moving actors in the foreground, ensuring that only static background contributes to pose estimation.
  • M-estimator for outlier rejection: An enhanced M-estimator with segmentation-based weights is used to improve robustness against outliers caused by dynamic scenes, lighting effects, or sensor noise.
  • Efficient GPU implementation: The entire pipeline, including minimization, warping, and linear system reduction, is implemented on a commodity GPU for real-time performance.

Datasets

The paper does not explicitly mention specific public datasets used for training. However, it states that experiments were conducted in a real television broadcasting studio and evaluated using sequences from the Freiburg sequences with known ground truth trajectory (freiburg2/desk and freiburg1/desk). The authors also mention using a Microsoft Kinect as the RGB-D sensor.

Key Insights and Ideas

  • Keyframes are crucial for drift elimination: The most significant insight is that by anchoring tracking to a static keyframe model, the system can overcome the inherent drift problem of purely incremental tracking methods, which is critical for long-term stability in studio environments.
  • Affordability through low-end hardware: The demonstration that a robust, real-time system can be achieved using a standard laptop and a low-end GPU is a major step towards democratizing advanced tracking technology.
  • Segmentation-based weighting for dynamic scenes: The ability to effectively reject moving actors by comparing current sensor data to a static keyframe model is a practical solution for real-world studio scenarios.
  • GPU optimization is key for real-time dense processing: The paper highlights the necessity of efficient GPU implementation to handle the computational demands of dense RGB-D data processing in real-time.

Key Results

  • Real-time performance: The system achieves real-time operation at 30Hz, with a computation time of 26ms per frame, leaving 7ms for rendering augmented graphics.
  • Drift reduction: The keyframe-based approach significantly reduces time-evolving drift compared to incremental dense tracking. For the freiburg2/desk sequence, the drift was 1.08cm/s, and for freiburg1/desk, it was 2.60cm/s.
  • Robustness: The system demonstrates robust tracking in a real television broadcasting studio, both with and without dynamic foreground elements.
  • Comparison with Kinfu: The proposed method shows comparable or better drift performance than Kinfu (a KinectFusion-based system) on specific sequences, particularly on planar surfaces where Kinfu struggles. The keyframe approach also offers lower memory consumption and is less sensitive to scene geometry variations.

Limitations and Future Work

  • Camera speed limitations: The minimization strategy is local, meaning it may not converge if the camera moves too quickly. The authors suggest that motion capture systems could provide initial guesses for poses to overcome this.
  • Scene texturing dependency: While the method is adaptable to different studio settings, its accuracy relies on sufficient scene texturing.
  • Kinect RGB camera quality: The low quality of the Microsoft Kinect RGB camera can lead to color saturation, which can be mitigated by calibrating the RGB-D sensor depth maps with a professional HD camera.
  • Future work: The authors plan to address practical issues for studio staff and cameramen using the system, calibrate the RGB-D sensor with a professional HD camera for better image quality, and investigate combining the best properties of their approach with KinectFusion.

Summary updated

BibTeX
@article{tykkala2014live,
  author = {Tommi Tykkälä and Andrew I. Comport and Joni-Kristian Kämäräinen and Hannu Hartikainen},
  title = {{Live RGB-D camera tracking for television production studios}},
  journal = {J. Vis. Commun. Image Represent.},
  year = {2014},
  doi = {10.1016/j.jvcir.2013.02.009},
}
Figure