Video Matting via Consistency-Regularized Graph Neural Networks

Tiantian Wang, Sifei Liu, Yapeng Tian, Kai Li, Ming-Hsuan Yang

Abstract

Video matting is a critical task in computer vision, requiring the estimation of precise foreground opacity (alpha mattes) across video sequences. Traditional approaches often suffer from temporal flickering and poor generalization to real-world scenes due to the reliance on synthetic training data. We present Consistency-Regularized Graph Neural Networks (CRGNN), a novel framework designed to address these challenges. By modeling video frames as nodes in a graph and employing a consistency-regularization loop for semi-supervised learning on unlabeled real-world data, CRGNN achieves superior temporal coherence and spatial accuracy.

Introduction

Estimating the alpha matte in video sequences is significantly more complex than binary segmentation. It requires capturing fine-grained details such as hair strands and motion blur while maintaining stability over time. Two primary obstacles hinder current progress:

  1. Temporal Inconsistency: Frame-by-frame processing often results in visual "jitter." While optical flow has been used to mitigate this, it is computationally expensive and sensitive to estimation errors.
  2. The Domain Gap: High-quality ground-truth labels for real-world video matting are scarce. Models trained exclusively on synthetic composites often fail to generalize to the complexities of real-world lighting and backgrounds.

Methodology

The CRGNN framework introduces three innovative components to overcome these hurdles:

Graph Neural Networks for Temporal Propagation

Instead of treating frames as independent entities, CRGNN constructs a fully connected graph where each node represents a video frame. Information is propagated across the temporal dimension using deformable alignment. This allows the model to leverage features from neighboring frames to resolve ambiguities caused by blur or occlusion in a specific frame, ensuring a more stable and accurate prediction.

Consistency Regularization

To bridge the domain gap, the authors implement a semi-supervised consistency-regularization loop. The model takes a predicted alpha matte and foreground from a real-world video, blends them with a novel random background, and re-processes this new composite. By enforcing consistency between the original and re-composited predictions, the network learns to handle real-world variations without requiring explicit ground-truth labels.

Adversarial Learning

A discriminator network is integrated to distinguish between synthetic and real-world frames. This adversarial setup encourages the generator to produce domain-agnostic results that appear realistic regardless of the background.

Datasets and Benchmarks

The study introduces three new datasets to facilitate research:

  • Composited Video Dataset: Large-scale synthetic datasets featuring humans and auxiliary categories (e.g., cats, plants) with high-quality mattes.
  • Real-world Video Dataset: A manually annotated set of 19 real-world videos (711 frames), providing a much-needed benchmark for practical evaluation.

Experimental Results

CRGNN demonstrates significant improvements over existing state-of-the-art methods. On synthetic human datasets, it achieved an MSE of 3.770, compared to 13.57 for previous leading video-based methods. Crucially, its temporal coherence (measured by MESSDdt) improved from 7.388 to 2.475, indicating a substantial reduction in flickering.

Visual comparison of alpha matte quality and re-compositing results. The CRGNN model preserves fine hair details more effectively than baselines, enabling high-quality re-compositing onto new backgrounds.

Qualitative analysis confirms that CRGNN excels at recovering intricate hair details and suppressing background noise, even in challenging real-world scenarios. The consistency-regularization loop allows the model to maintain high fidelity when the subject is placed in entirely new environments.

Discussion and Future Work

While CRGNN sets a new standard for accuracy and consistency, its current processing speed of 1 fps on an NVIDIA 2080 Ti limits its use in real-time applications like live video conferencing. Additionally, the model still relies on "pseudo-trimaps" for guidance. Future research will focus on optimizing graph structures for efficiency and developing fully trimap-free high-resolution matting systems.

Conclusion

CRGNN effectively addresses the dual challenges of temporal inconsistency and the synthetic-to-real domain gap in video matting. By combining GNN-based temporal modeling with a self-supervised consistency loop, it provides a robust solution for high-quality video editing and compositing.

Summary updated

Abstract

Learning temporally consistent foreground opacity from videos, i.e., video matting, has drawn great attention due to the blossoming of video conferencing. Previous approaches are built on top of image matting models, which fail in maintaining the temporal coherence when being adapted to videos. They either utilize the optical flow to smooth frame-wise prediction, where the performance is dependent on the selected optical flow model; or naively combine feature maps from multiple frames, which does not model well the correspondence of pixels in adjacent frames. In this paper, we propose to enhance the temporal coherence by Consistency-Regularized Graph Neural Networks (CRGNN) with the aid of a synthesized video matting dataset. CRGNN utilizes Graph Neural Networks (GNN) to relate adjacent frames such that pixels or regions that are incorrectly predicted in one frame can be corrected by leveraging information from its neighboring frames. To generalize our model from synthesized videos to real-world videos, we propose a consistency regularization technique to enforce the consistency on the alpha and foreground when blending them with different backgrounds. To evaluate the efficacy of CRGNN, we further collect a real-world dataset with annotated alpha mattes. Compared with state-of-the-art methods that require hand-crafted trimaps or backgrounds for modeling training, CRGNN generates favorably results with the help of unlabeled real training dataset. The source code and datasets are available at https://github.com/TiantianWang/VideoMattingCRGNN.git.
BibTeX
@inproceedings{wang2021video,
  author = {Tiantian Wang and Sifei Liu and Yapeng Tian and Kai Li and Ming-Hsuan Yang},
  title = {{Video Matting via Consistency-Regularized Graph Neural Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00486},
}
Figure