Abstract
We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance. Our method is much lighter than previous approaches and can process 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080Ti GPU. Unlike most existing methods that perform video matting frame-by-frame as independent images, our method uses a recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and matting quality. Furthermore, we propose a novel training strategy that enforces our network on both matting and segmentation objectives. This significantly improves our model's robustness. Our method does not require any auxiliary inputs such as a trimap or a pre-captured background image, so it can be widely applied to existing human matting applications.
TL;DR
This paper introduces a real-time, high-resolution human video matting network that uses a recurrent architecture to leverage temporal information across frames. By jointly training on both matting and semantic segmentation objectives, the model achieves state-of-the-art robustness and temporal coherence without needing auxiliary inputs like trimaps or pre-captured backgrounds, all while processing 4K video at 76 FPS on a single GPU.
What Problem Does This Paper Address?
Video matting is the process of precisely separating a foreground subject (like a human) from the background to create an alpha matte and foreground color layer. It is the core technology behind virtual backgrounds in video conferencing and green-screen-free video editing.
However, most existing neural matting methods process videos frame-by-frame as independent images. This ignores the rich temporal information inherent in video, leading to annoying edge flickering and artifacts. Furthermore, many high-quality matting methods require auxiliary inputs—such as a manually drawn "trimap" or a pre-captured image of the empty background—which are highly impractical for fully automatic, real-time applications where the camera or background might move.
Methods
The authors propose a recurrent neural network architecture designed specifically to exploit the temporal nature of video, entirely removing the need for auxiliary inputs.
- Feature-Extraction Encoder: The network uses a lightweight MobileNetV3-Large backbone combined with an LR-ASPP module to extract multi-scale spatial features from individual frames.
- Recurrent Decoder: The decoder employs ConvGRU layers at multiple scales to aggregate temporal information. To save compute, a split-and-concatenate design is used: ConvGRU is applied to only half of the channels to focus on temporal aggregation, while the other half forwards spatial features specific to the current frame.
- High-Resolution Upsampling: To achieve 4K and HD resolution in real-time, the core encoder-decoder operates at a lower resolution. A Deep Guided Filter (DGF) module is then used to efficiently upsample the predictions back to high resolution.
- Joint Training Strategy: To combat overfitting to synthetic lighting mismatches, the authors interleave matting training with a semantic segmentation objective using real-world images from datasets like YouTubeVIS, COCO, and SPD.
Key Insights and Ideas
- Memory is Magic: The recurrent hidden states learn to reconstruct the occluded background as the camera or subject moves. By "remembering" the background, the network simplifies the matting task in future frames. It even learns to "forget" this memory automatically during hard shot cuts.
- Segmentation as Regularization: Forcing the network to solve standard semantic segmentation on real images acts as a powerful regularizer. It prevents the model from overfitting to synthetic artifacts, making it highly robust to complex, real-world scenes.
- Temporal Coherence: Seeing multiple frames doesn't just help with accuracy; it acts as a massive boost to perceptual quality. By referring to previous frames, the model avoids the high-frequency jitter (flickering) that plagues frame-by-frame models.
Key Results
The model is incredibly lightweight, containing only 3.75 million parameters (about 58% of the size of the competing state-of-the-art MODNet). It achieves 104 FPS at HD (1920x1080) and 76 FPS at 4K (3840x2160) on an Nvidia GTX 1080Ti.
It outperforms existing auxiliary-free methods (like MODNet) and background-based methods (like BGMv2) across multiple metrics (MAD, MSE, Spatial Gradient). The recurrent architecture achieves significantly better temporal stability (measured by dtSSD) compared to frame-by-frame methods, producing smooth, coherent mattes even on fast-moving body parts like hands and hair.
Limitations and Future Work
Because the model is trained specifically to matte humans, it struggles when there are other people in the background. The network can become confused about who the primary subject of interest is. Additionally, highly cluttered or complex scenes can degrade the accuracy of the alpha matte. Future work includes exploring larger backbones like ResNet50 for server-side applications where higher performance is prioritized over speed.
Summary updated