Robust High-Resolution Video Matting with Temporal Guidance

Shanchuan Lin, Linjie Yang, Imran Saleemi, Roni Sengupta

TL;DR

This paper introduces a real-time, high-resolution human video matting network that uses a recurrent architecture to leverage temporal information across frames. By jointly training on both matting and semantic segmentation objectives, the model achieves state-of-the-art robustness and temporal coherence without needing auxiliary inputs like trimaps or pre-captured backgrounds, all while processing 4K video at 76 FPS on a single GPU.

What Problem Does This Paper Address?

Video matting is the process of precisely separating a foreground subject (like a human) from the background to create an alpha matte and foreground color layer. It is the core technology behind virtual backgrounds in video conferencing and green-screen-free video editing.

However, most existing neural matting methods process videos frame-by-frame as independent images. This ignores the rich temporal information inherent in video, leading to annoying edge flickering and artifacts. Furthermore, many high-quality matting methods require auxiliary inputs—such as a manually drawn "trimap" or a pre-captured image of the empty background—which are highly impractical for fully automatic, real-time applications where the camera or background might move.

Methods

The authors propose a recurrent neural network architecture designed specifically to exploit the temporal nature of video, entirely removing the need for auxiliary inputs.

  • Feature-Extraction Encoder: The network uses a lightweight MobileNetV3-Large backbone combined with an LR-ASPP module (a design borrowed from semantic segmentation) to extract multi-scale spatial features from individual frames.
  • Recurrent Decoder: The decoder employs ConvGRU (Convolutional Gated Recurrent Unit) layers at multiple scales to aggregate temporal information. To save compute, the authors use a clever split-and-concatenate design: ConvGRU is applied to only half of the channels to focus on temporal aggregation, while the other half forwards spatial features specific to the current frame.
  • High-Resolution Upsampling: To achieve 4K and HD resolution in real-time, the core encoder-decoder operates at a lower resolution. A Deep Guided Filter (DGF) module is then used to efficiently upsample the predictions back to high resolution.
  • Joint Training Strategy: Matting datasets rely on synthetically compositing foregrounds onto backgrounds, which can cause models to overfit to lighting mismatches. To combat this, the authors interleave matting training with a semantic segmentation objective using real-world images.

Datasets

The model is trained and evaluated on a mix of synthetic matting datasets and real-world segmentation datasets:

  • Matting Datasets: VideoMatte240K (VM), Distinctions-646 (D646), and Adobe Image Matting (AIM). These provide ground-truth alpha mattes.
  • Segmentation Datasets: YouTubeVIS, COCO, and SPD. These provide real-world images with human segmentation masks to improve the model's semantic robustness.
  • Backgrounds: HD background videos (featuring dynamic elements like moving cars) and crawled indoor/outdoor images were used for compositing the matting training data.

Key Insights and Ideas

  • Memory is Magic: The recurrent hidden states actually learn to reconstruct the occluded background as the camera or subject moves. By "remembering" what the background looks like behind the person, the network drastically simplifies its own matting task in future frames. It even learns to "forget" this memory automatically during hard shot cuts.
  • Segmentation as Regularization: Forcing the network to solve standard semantic segmentation on real images acts as a powerful regularizer. It prevents the model from overfitting to the synthetic artifacts of matting datasets, making it highly robust to complex, real-world scenes.
  • Temporal Coherence: Seeing multiple frames doesn't just help with accuracy; it acts as a massive boost to perceptual quality. By referring to previous frames, the model avoids the high-frequency jitter (flickering) that plagues frame-by-frame models.

Key Results

  • Speed and Efficiency: The model is incredibly lightweight, containing only 3.75 million parameters (about 58% of the size of the competing state-of-the-art MODNet). It achieves 104 FPS at HD (1920x1080) and 76 FPS at 4K (3840x2160) on an Nvidia GTX 1080Ti.
  • State-of-the-Art Quality: It outperforms existing auxiliary-free methods (like MODNet) and background-based methods (like BGMv2) across multiple metrics (MAD, MSE, Spatial Gradient).
  • Flicker Reduction: The recurrent architecture achieves significantly better temporal stability (measured by dtSSD) compared to frame-by-frame methods, producing smooth, coherent mattes even on fast-moving body parts like hands and hair.

Limitations and Future Work

  • Ambiguous Targets: Because the model is trained specifically to matte humans, it struggles when there are other people in the background. The network can become confused about who the primary subject of interest is.
  • Complex Backgrounds: While robust, the model still favors simpler backgrounds. Highly cluttered or complex scenes can degrade the accuracy of the alpha matte.
  • Future Work: The authors note that for server-side applications where compute is less constrained, the lightweight MobileNetV3 backbone can be swapped for a larger ResNet50 backbone. This trades some speed for even higher matting performance.

Summary updated

Abstract

We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance. Our method is much lighter than previous approaches and can process 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080Ti GPU. Unlike most existing methods that perform video matting frame-by-frame as independent images, our method uses a recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and matting quality. Furthermore, we propose a novel training strategy that enforces our network on both matting and segmentation objectives. This significantly improves our model’s robustness. Our method does not require any auxiliary inputs such as a trimap or a pre-captured background image, so it can be widely applied to existing human matting applications. Our code is available at https://peterl1n.github.io/RobustVideoMatting/
BibTeX
@inproceedings{lin2022robust,
  author = {Shanchuan Lin and Linjie Yang and Imran Saleemi and Roni Sengupta},
  title = {{Robust High-Resolution Video Matting with Temporal Guidance}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=YcOVHajN-m9},
}

Loading related content...

Figure