TL;DR
OmnimatteZero is the first training-free, real-time approach for "omnimatte"—decomposing video into background and foreground layers that include complex effects like shadows and reflections. By leveraging pre-trained video diffusion models and novel attention guidance modules, it achieves state-of-the-art results in just 0.04 seconds per frame, eliminating the hours of per-video optimization required by previous methods.
What Problem Does This Paper Address?
Video editing tasks like object removal or background replacement are notoriously difficult because objects don't exist in isolation; they cast shadows, create reflections, and perturb their environment. The "omnimatte" task aims to extract these objects along with their associated effects.
Until now, researchers faced a trade-off:
- High Quality, Low Speed: Traditional omnimatte methods require hours of per-video optimization or extensive training on hand-crafted datasets.
- Zero-Shot, Low Consistency: Adapting image-based inpainting models to video often results in "flickering" or incoherent "blobs" because these models lack temporal awareness. They can remove an object but fail to reconstruct a consistent background or handle the subtle traces (like a shadow) the object left behind.
OmnimatteZero fills this gap by providing a generative, training-free framework that is both temporally consistent and fast enough for real-time applications.
Methods
The authors build OmnimatteZero on top of off-the-shelf video diffusion transformers (like LTX-Video). Their core innovation lies in Attention Guidance, which steers the diffusion process without changing the model's weights.
1. Temporal and Spatial Attention Guidance (TAG & SAG)
To ensure the background looks the same across frames, the model needs to "look" at neighboring frames to see what was behind the object.
- TAG: Uses a point-tracking framework (TAP-Net) to identify background patches in adjacent frames. It then forces the self-attention mechanism to prioritize these known background patches when reconstructing a masked area.
- SAG: Operates within a single frame, guiding the model to use surrounding background textures to fill in gaps when temporal information is missing.
2. Effect-Aware Masking
Standard masks only cover the object. OmnimatteZero uses the self-attention maps of the diffusion model itself to identify "footprints"—shadows and reflections. Because video models are trained on motion, they naturally group objects with their effects (a concept in psychology called "common fate"). The system automatically expands the user's mask to include these effects for a cleaner removal.
3. Latent Arithmetic and Pixel Injection
For object extraction, the authors use a simple but effective trick: they subtract the latent representation of the "clean" background from the latent representation of the original video. To maintain high fidelity, they "inject" original pixels back into the foreground layer, ensuring the extracted object looks exactly like the original while retaining its semi-transparent shadows.
Datasets
The authors evaluated OmnimatteZero on two primary benchmarks:
- Movies: A dataset of real-world videos featuring complex motions and backgrounds.
- Kubric: A synthetic dataset that provides perfect ground-truth background layers for quantitative evaluation.
Key Insights and Ideas
- Motion as a Grouper: One of the most interesting findings is that video diffusion models are significantly better at "seeing" shadows than image models. Because shadows move in sync with the object, the spatio-temporal attention mechanism naturally links them together.
- Guidance over Optimization: You don't need to retrain a massive model to do video matting. By simply manipulating the attention scores during the denoising process, you can achieve better results than methods that spend hours optimizing for a single clip.
- Real-Time Feasibility: By operating entirely within the inference pass of a fast transformer-based model, the method hits 24 FPS on an A100 GPU, making it orders of magnitude faster than prior work.
Key Results
- Speed: OmnimatteZero runs at 0.04s per frame, compared to ~9s for recent generative competitors and hours of pre-processing for optimization-based methods.
- Accuracy: On the Kubric dataset, it achieved a PSNR of 45.55, significantly higher than the previous state-of-the-art (Generative Omnimatte at 44.07).
- Qualitative Superiority: As shown in Figure 1, the method seamlessly handles complex cases like removing a cat's reflection from a mirror or correcting the "bending" of a trampoline after a jumper is removed.
Limitations and Future Work
Despite its performance, the authors note a few hurdles:
- VAE Artifacts: Because the method relies on a Variational Autoencoder (VAE) for compression, the final reconstructed video may have very slight deviations in color or texture compared to the original raw pixels.
- Tracking Dependency: The quality of the temporal guidance is only as good as the underlying point-tracker (TAP-Net). In cases of extreme occlusion where the background is never visible in any frame, the model must revert to "hallucinating" content.
- Model Capacity: The fidelity of the inpainting is ultimately capped by the pre-trained video diffusion model's knowledge. If the model hasn't seen a specific type of complex scene before, it may struggle to reconstruct it accurately.
Future work may involve integrating more robust tracking mechanisms and leveraging even larger, more capable video foundation models as they become available.
Summary updated