Abstract
Video matting is a critical task in computer vision, requiring the estimation of precise foreground opacity (alpha mattes) across video sequences. Traditional approaches often suffer from temporal flickering and poor generalization to real-world scenes due to the reliance on synthetic training data. We present Consistency-Regularized Graph Neural Networks (CRGNN), a novel framework designed to address these challenges. By modeling video frames as nodes in a graph and employing a consistency-regularization loop for semi-supervised learning on unlabeled real-world data, CRGNN achieves superior temporal coherence and spatial accuracy.
Introduction
Estimating the alpha matte in video sequences is significantly more complex than binary segmentation. It requires capturing fine-grained details such as hair strands and motion blur while maintaining stability over time. Two primary obstacles hinder current progress:
- Temporal Inconsistency: Frame-by-frame processing often results in visual "jitter." While optical flow has been used to mitigate this, it is computationally expensive and sensitive to estimation errors.
- The Domain Gap: High-quality ground-truth labels for real-world video matting are scarce. Models trained exclusively on synthetic composites often fail to generalize to the complexities of real-world lighting and backgrounds.
Methodology
The CRGNN framework introduces three innovative components to overcome these hurdles:
Graph Neural Networks for Temporal Propagation
Instead of treating frames as independent entities, CRGNN constructs a fully connected graph where each node represents a video frame. Information is propagated across the temporal dimension using deformable alignment. This allows the model to leverage features from neighboring frames to resolve ambiguities caused by blur or occlusion in a specific frame, ensuring a more stable and accurate prediction.
Consistency Regularization
To bridge the domain gap, the authors implement a semi-supervised consistency-regularization loop. The model takes a predicted alpha matte and foreground from a real-world video, blends them with a novel random background, and re-processes this new composite. By enforcing consistency between the original and re-composited predictions, the network learns to handle real-world variations without requiring explicit ground-truth labels.
Adversarial Learning
A discriminator network is integrated to distinguish between synthetic and real-world frames. This adversarial setup encourages the generator to produce domain-agnostic results that appear realistic regardless of the background.
Datasets and Benchmarks
The study introduces three new datasets to facilitate research:
- Composited Video Dataset: Large-scale synthetic datasets featuring humans and auxiliary categories (e.g., cats, plants) with high-quality mattes.
- Real-world Video Dataset: A manually annotated set of 19 real-world videos (711 frames), providing a much-needed benchmark for practical evaluation.
Experimental Results
CRGNN demonstrates significant improvements over existing state-of-the-art methods. On synthetic human datasets, it achieved an MSE of 3.770, compared to 13.57 for previous leading video-based methods. Crucially, its temporal coherence (measured by MESSDdt) improved from 7.388 to 2.475, indicating a substantial reduction in flickering.
Qualitative analysis confirms that CRGNN excels at recovering intricate hair details and suppressing background noise, even in challenging real-world scenarios. The consistency-regularization loop allows the model to maintain high fidelity when the subject is placed in entirely new environments.
Discussion and Future Work
While CRGNN sets a new standard for accuracy and consistency, its current processing speed of 1 fps on an NVIDIA 2080 Ti limits its use in real-time applications like live video conferencing. Additionally, the model still relies on "pseudo-trimaps" for guidance. Future research will focus on optimizing graph structures for efficiency and developing fully trimap-free high-resolution matting systems.
Conclusion
CRGNN effectively addresses the dual challenges of temporal inconsistency and the synthetic-to-real domain gap in video matting. By combining GNN-based temporal modeling with a self-supervised consistency loop, it provides a robust solution for high-quality video editing and compositing.
Summary updated