OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models

Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, Rami Ben-Ari

TL;DR

OmnimatteZero is the first training-free, real-time approach for "omnimatte"—decomposing video into background and foreground layers that include complex effects like shadows and reflections. By leveraging pre-trained video diffusion models and novel attention guidance modules, it achieves state-of-the-art results in just 0.04 seconds per frame, eliminating the hours of per-video optimization required by previous methods.

What Problem Does This Paper Address?

Video editing tasks like object removal or background replacement are notoriously difficult because objects don't exist in isolation; they cast shadows, create reflections, and perturb their environment. The "omnimatte" task aims to extract these objects along with their associated effects.

Until now, researchers faced a trade-off:

  1. High Quality, Low Speed: Traditional omnimatte methods require hours of per-video optimization or extensive training on hand-crafted datasets.
  2. Zero-Shot, Low Consistency: Adapting image-based inpainting models to video often results in "flickering" or incoherent "blobs" because these models lack temporal awareness. They can remove an object but fail to reconstruct a consistent background or handle the subtle traces (like a shadow) the object left behind.

OmnimatteZero fills this gap by providing a generative, training-free framework that is both temporally consistent and fast enough for real-time applications.

Fig. 2. Comparison of object removal results using (a) vanilla image in- painting extended to video, (b) Vid2Vid zero-shot inpainting, and (c) our guidance-based approach. Vanilla methods fail to maintain temporal con- sistency or clean background reconstruction, while our method achieves coherent, artifact-free inpainting across frames.

Methods

The authors build OmnimatteZero on top of off-the-shelf video diffusion transformers (like LTX-Video). Their core innovation lies in Attention Guidance, which steers the diffusion process without changing the model's weights.

1. Temporal and Spatial Attention Guidance (TAG & SAG)

To ensure the background looks the same across frames, the model needs to "look" at neighboring frames to see what was behind the object.

  • TAG: Uses a point-tracking framework (TAP-Net) to identify background patches in adjacent frames. It then forces the self-attention mechanism to prioritize these known background patches when reconstructing a masked area.
  • SAG: Operates within a single frame, guiding the model to use surrounding background textures to fill in gaps when temporal information is missing.

2. Effect-Aware Masking

Standard masks only cover the object. OmnimatteZero uses the self-attention maps of the diffusion model itself to identify "footprints"—shadows and reflections. Because video models are trained on motion, they naturally group objects with their effects (a concept in psychology called "common fate"). The system automatically expands the user's mask to include these effects for a cleaner removal.

3. Latent Arithmetic and Pixel Injection

For object extraction, the authors use a simple but effective trick: they subtract the latent representation of the "clean" background from the latent representation of the original video. To maintain high fidelity, they "inject" original pixels back into the foreground layer, ensuring the extracted object looks exactly like the original while retaining its semi-transparent shadows.

Fig. 3. Overview of our Object Removal strategy in OmnimatteZero. (a) We first identify potential background correspondences across frames. (b) Temporal Attention Guidance (TAG): Temporal attention scores between a foreground point and its background correspondences are replaced with the average attention between all background pairs, promoting consistent inpainting across time. (c) Spatial Attention Guidance (SAG): Within a frame, the attention from a foreground point to nearby background points is adjusted to reflect the mean attention among background points themselves, improving inpainting quality when temporal context is unavailable.

Datasets

The authors evaluated OmnimatteZero on two primary benchmarks:

  • Movies: A dataset of real-world videos featuring complex motions and backgrounds.
  • Kubric: A synthetic dataset that provides perfect ground-truth background layers for quantitative evaluation.

Key Insights and Ideas

  • Motion as a Grouper: One of the most interesting findings is that video diffusion models are significantly better at "seeing" shadows than image models. Because shadows move in sync with the object, the spatio-temporal attention mechanism naturally links them together.
  • Guidance over Optimization: You don't need to retrain a massive model to do video matting. By simply manipulating the attention scores during the denoising process, you can achieve better results than methods that spend hours optimizing for a single clip.
  • Real-Time Feasibility: By operating entirely within the inference pass of a fast transformer-based model, the method hits 24 FPS on an A100 GPU, making it orders of magnitude faster than prior work.

Key Results

  • Speed: OmnimatteZero runs at 0.04s per frame, compared to ~9s for recent generative competitors and hours of pre-processing for optimization-based methods.
  • Accuracy: On the Kubric dataset, it achieved a PSNR of 45.55, significantly higher than the previous state-of-the-art (Generative Omnimatte at 44.07).
  • Qualitative Superiority: As shown in Figure 1, the method seamlessly handles complex cases like removing a cat's reflection from a mirror or correcting the "bending" of a trampoline after a jumper is removed.
Fig. 1. OmnimatteZero is the first training-free generative approach for Omnimatte, leveraging pre-trained video diffusion models to achieve object removal, extraction, and seamless layer compositions in just 0.04 sec/frame (on an A100 GPU). See video examples in the supplemental material.

Limitations and Future Work

Despite its performance, the authors note a few hurdles:

  • VAE Artifacts: Because the method relies on a Variational Autoencoder (VAE) for compression, the final reconstructed video may have very slight deviations in color or texture compared to the original raw pixels.
  • Tracking Dependency: The quality of the temporal guidance is only as good as the underlying point-tracker (TAP-Net). In cases of extreme occlusion where the background is never visible in any frame, the model must revert to "hallucinating" content.
  • Model Capacity: The fidelity of the inpainting is ultimately capped by the pre-trained video diffusion model's knowledge. If the model hasn't seen a specific type of complex scene before, it may struggle to reconstruct it accurately.

Future work may involve integrating more robust tracking mechanisms and leveraging even larger, more capable video foundation models as they become available.

Summary updated

Abstract

In Omnimatte, one aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization. In this paper, we present OmnimatteZero, a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos. These are accomplished by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. To overcome this, we introduce temporal and spatial attention guidance modules that steer the diffusion process for accurate object removal and temporally consistent background reconstruction. We further show that self-attention maps capture information about the object and its footprints and use them to inpaint the object's effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos. Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime.
BibTeX
@inproceedings{samuel2025omnimattezero,
  author = {Dvir Samuel and Matan Levy and Nir Darshan and Gal Chechik and Rami Ben-Ari},
  title = {{OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models}},
  booktitle = {SIGGRAPH Asia},
  year = {2025},
  doi = {10.1145/3757377.3763917},
}
Figure