FactorMatte: Redefining Video Matting for Re-Composition Tasks

Zeqi Gu, Wenqi Xian, Noah Snavely, Abe Davis

Abstract

FactorMatte redefines video matting as a "counterfactual synthesis" problem, moving beyond simple foreground-background separation. By using a Bayesian framework with conditional priors, it successfully disentangles complex scene interactions—like splashes, shadows, and reflections—allowing for high-quality video re-composition that previous methods couldn't handle.

Introduction: The Independence Assumption

Traditional video matting relies on the classic compositing equation: $I = \alpha F + (1-\alpha)B$. This model assumes that the foreground ($F$) and background ($B$) are independent. However, in the real world, they rarely are. For example, a child jumping into a puddle creates a splash; the splash is a background interaction physically caused by the foreground object.

Standard matting often "bakes" these interactions into the wrong layer, leading to artifacts during re-composition. FactorMatte addresses this by factoring complex interactions into independent, "counterfactual" components, essentially asking: "What would the background look like if the foreground object were never there?"

Methodology

FactorMatte treats matting as a Maximum A Posteriori (MAP) estimation problem, moving from marginal priors to conditional priors ($P(F|B)$ and $P(B|F)$).

The FactorMatte pipeline decomposes video into Foreground, Environment, and Background Residual layers.

The system utilizes a Decomposition Network ($N_D$) optimized per-video to factor the input into three distinct layers:

  1. Environment Layer: The static background.
  2. Residual Layer: Irregular background changes (e.g., splashes, shadows).
  3. Foreground Layer: The primary moving object.

Training involves three stages, culminating in Conditional Approximation, where the network is trained adversarially against patch-based discriminators. These discriminators learn to distinguish between "real" augmented background patches and the network's outputs, forcing the generation of physically plausible, independent layers.

Key Results and Applications

FactorMatte produces significantly cleaner mattes for fine details and handles semi-transparent interactions better than previous state-of-the-art methods like Omnimatte. On the CDW-2014 background subtraction benchmark, it achieved an AUC of 0.934.

A re-composited video result showing a new character interacting with the original environment's splash.

The framework enables advanced downstream tasks such as "Color Pop" (modifying the color of light beams) and "Inpainting" (removing objects while preserving their environmental effects).

Limitations

The primary drawback is computational cost; per-video optimization makes FactorMatte approximately 3.5x slower than Omnimatte. Additionally, the model can struggle with visual ambiguity when foreground and background share very similar textures or colors.

Conclusion

By framing matting as counterfactual synthesis, FactorMatte provides a robust solution for complex video editing tasks, successfully disentangling the intricate dance between objects and their environments.

Summary updated

BibTeX
@article{gu2023factormatte,
  author = {Zeqi Gu and Wenqi Xian and Noah Snavely and Abe Davis},
  title = {{FactorMatte: Redefining Video Matting for Re-Composition Tasks}},
  journal = {ACM Trans. Graphics},
  year = {2023},
  doi = {10.1145/3592423},
}
Figure