Color-aware Deep Temporal Backdrop Duplex Matting System

Hendrik Hachmann, Bodo Rosenhahn

Introduction

In professional film production, "matting"—the process of isolating a foreground subject from its background—remains a fundamental yet challenging task. While traditional chroma keying (green screening) has been the industry standard for decades, it is plagued by two persistent issues: color spill and the labor-intensive requirement for manual trimaps. Color spill occurs when the vibrant background color reflects onto the subject, creating unnatural fringes, while trimaps require artists to manually define foreground and background regions for every frame.

This paper presents a "temporal backdrop duplex" system that leverages a hardware-software co-design to automate high-quality matting. By utilizing a high-speed LED wall and a specialized dual-decoder neural network, the system eliminates the need for trimaps while effectively suppressing color spill and handling dynamic motion.

The Temporal Backdrop Duplex System

Hardware Configuration

The core of the system is a synchronized hardware setup consisting of a high-speed global shutter camera (operating at 100 fps) and an FPGA-controlled LED wall. The LED wall rapidly cycles through a specific sequence: Green $\rightarrow$ VFX Content $\rightarrow$ Blue $\rightarrow$ VFX Content.

The camera is precisely synchronized to capture only the green and blue frames. This "time duplexing" allows the system to obtain two different background color references for essentially the same scene. A significant advantage of this approach is that the actor primarily sees the "VFX Content" frames, creating a more immersive and comfortable environment than a static green screen.

Neural Architecture

The software component features a one-encoder, dual-decoder neural network architecture. The network takes two consecutive patches from the video (one with a green background and one with a blue background) as input.

  1. Alpha Decoder: Predicts the alpha matte, representing the transparency of each pixel.
  2. Color Decoder: Predicts the "clean" foreground RGB colors, effectively reconstructing the subject's true appearance without background contamination.

Addressing Motion and Color Spill

Motion Augmentation

A primary challenge in temporal matting is subject movement between frames. If an actor moves quickly between the green and blue captures, a naive network might produce "ghosting" artifacts. To solve this, the authors implemented a novel motion augmentation training strategy. By simulating movement between input frames during the training phase, the network learns to account for temporal shifts, ensuring the alpha matte remains sharp even during rapid action.

Latent Space Spill Suppression

Traditional methods often use simple heuristic filters to remove green fringes, which can desaturate the subject's actual colors. This system instead utilizes a "color-aware" latent space. The network learns the underlying distribution of foreground colors, allowing it to mathematically neutralize spill. By blending the original image with the predicted clean foreground colors based on alpha confidence, the system "bleaches" out background leakage while preserving the integrity of fine details like hair and translucent materials.

Experimental Results

The proposed system, referred to as ours$_{ma}$ (motion augmented), was evaluated against state-of-the-art (SOTA) trimap-based methods, including FBA and SIM, using standard benchmarks like Adobe Composition-1k and a custom 5,000-frame background dataset.

Quantitative Performance

On datasets such as Alex, Dmitriy, and Castle, the system achieved SOTA-level performance, reaching an MS-SSIM of 0.997 and a PSNR of 44.8. Notably, it often outperformed methods that utilized ground-truth trimaps, despite our system operating without any manual input.

Qualitative Stability

In the "Troll" sequence—a challenging test involving a 3D-printed model with fine yellow and green hair—the system demonstrated superior suppression of color spill and chromatic aberration. While traditional methods like BSM and CIM exhibited double contours and visible seams, this system produced stable, flicker-free results. It achieved the lowest Mean Alpha Deviation (MAD), confirming its high temporal consistency.

Conclusion and Future Work

The Color-aware Deep Temporal Backdrop Duplex system represents a significant step toward fully automated, production-quality matting. By combining high-speed LED synchronization with motion-aware deep learning, it solves the dual problems of color spill and manual labor.

Future research will focus on explicitly modeling lens dispersion to address chromatic aberration more precisely at the architectural level. Additionally, the authors aim to adapt the system for view-dependent backdrops, allowing the LED wall to display perspective-correct VFX content (similar to virtual production techniques used in The Mandalorian) while maintaining the dual-color matting functionality.

Summary updated

BibTeX
@inproceedings{hachmann2023color,
  author = {Hendrik Hachmann and Bodo Rosenhahn},
  title = {{Color-aware Deep Temporal Backdrop Duplex Matting System}},
  booktitle = {MMSys},
  year = {2023},
  doi = {10.1145/3587819.3590973},
}
Figure