TL;DR
This paper introduces a real-time, high-resolution human video matting network that uses a recurrent architecture to leverage temporal information across frames. By jointly training on both matting and semantic segmentation objectives, the model achieves state-of-the-art robustness and temporal coherence without needing auxiliary inputs like trimaps or pre-captured backgrounds, all while processing 4K video at 76 FPS on a single GPU.
What Problem Does This Paper Address?
Video matting is the process of precisely separating a foreground subject (like a human) from the background to create an alpha matte and foreground color layer. It is the core technology behind virtual backgrounds in video conferencing and green-screen-free video editing.
However, most existing neural matting methods process videos frame-by-frame as independent images. This ignores the rich temporal information inherent in video, leading to annoying edge flickering and artifacts. Furthermore, many high-quality matting methods require auxiliary inputs—such as a manually drawn "trimap" or a pre-captured image of the empty background—which are highly impractical for fully automatic, real-time applications where the camera or background might move.
Methods
The authors propose a recurrent neural network architecture designed specifically to exploit the temporal nature of video, entirely removing the need for auxiliary inputs.
- Feature-Extraction Encoder: The network uses a lightweight MobileNetV3-Large backbone combined with an LR-ASPP module (a design borrowed from semantic segmentation) to extract multi-scale spatial features from individual frames.
- Recurrent Decoder: The decoder employs ConvGRU (Convolutional Gated Recurrent Unit) layers at multiple scales to aggregate temporal information. To save compute, the authors use a clever split-and-concatenate design: ConvGRU is applied to only half of the channels to focus on temporal aggregation, while the other half forwards spatial features specific to the current frame.
- High-Resolution Upsampling: To achieve 4K and HD resolution in real-time, the core encoder-decoder operates at a lower resolution. A Deep Guided Filter (DGF) module is then used to efficiently upsample the predictions back to high resolution.
- Joint Training Strategy: Matting datasets rely on synthetically compositing foregrounds onto backgrounds, which can cause models to overfit to lighting mismatches. To combat this, the authors interleave matting training with a semantic segmentation objective using real-world images.
Datasets
The model is trained and evaluated on a mix of synthetic matting datasets and real-world segmentation datasets:
- Matting Datasets: VideoMatte240K (VM), Distinctions-646 (D646), and Adobe Image Matting (AIM). These provide ground-truth alpha mattes.
- Segmentation Datasets: YouTubeVIS, COCO, and SPD. These provide real-world images with human segmentation masks to improve the model's semantic robustness.
- Backgrounds: HD background videos (featuring dynamic elements like moving cars) and crawled indoor/outdoor images were used for compositing the matting training data.
Key Insights and Ideas
- Memory is Magic: The recurrent hidden states actually learn to reconstruct the occluded background as the camera or subject moves. By "remembering" what the background looks like behind the person, the network drastically simplifies its own matting task in future frames. It even learns to "forget" this memory automatically during hard shot cuts.
- Segmentation as Regularization: Forcing the network to solve standard semantic segmentation on real images acts as a powerful regularizer. It prevents the model from overfitting to the synthetic artifacts of matting datasets, making it highly robust to complex, real-world scenes.
- Temporal Coherence: Seeing multiple frames doesn't just help with accuracy; it acts as a massive boost to perceptual quality. By referring to previous frames, the model avoids the high-frequency jitter (flickering) that plagues frame-by-frame models.
Key Results
- Speed and Efficiency: The model is incredibly lightweight, containing only 3.75 million parameters (about 58% of the size of the competing state-of-the-art MODNet). It achieves 104 FPS at HD (1920x1080) and 76 FPS at 4K (3840x2160) on an Nvidia GTX 1080Ti.
- State-of-the-Art Quality: It outperforms existing auxiliary-free methods (like MODNet) and background-based methods (like BGMv2) across multiple metrics (MAD, MSE, Spatial Gradient).
- Flicker Reduction: The recurrent architecture achieves significantly better temporal stability (measured by dtSSD) compared to frame-by-frame methods, producing smooth, coherent mattes even on fast-moving body parts like hands and hair.
Limitations and Future Work
- Ambiguous Targets: Because the model is trained specifically to matte humans, it struggles when there are other people in the background. The network can become confused about who the primary subject of interest is.
- Complex Backgrounds: While robust, the model still favors simpler backgrounds. Highly cluttered or complex scenes can degrade the accuracy of the alpha matte.
- Future Work: The authors note that for server-side applications where compute is less constrained, the lightweight MobileNetV3 backbone can be swapped for a larger ResNet50 backbone. This trades some speed for even higher matting performance.
Summary updated