Video Object Matting via Hierarchical Space-Time Semantic Guidance

Yumeng Wang, Bo Xu, Ziwen Li, Han Huang, Cheng Lu, Yandong GuoWACV 2023 · DOI

Abstract

Video Object Matting (VOM) is a critical task for high-quality video editing, yet it remains challenging due to the need for precise temporal consistency and the labor-intensive nature of creating per-frame trimaps. We introduce Hierarchical Space-Time Semantic Guidance (HSTSG), a novel framework that produces temporally stable alpha mattes for entire video sequences using only a single coarse mask from the first frame. Our approach integrates a Hierarchical Memory Matching Module (HMMM) to capture multi-scale correspondences and a Cross-Frame Attention Refinement (CFAR) module to ensure smooth transitions. Evaluated on the VideoMatting108 and DVM benchmarks, HSTSG significantly outperforms existing state-of-the-art methods in both accuracy and temporal coherence.

Introduction

Extracting foreground objects with semi-transparent details—such as hair, fur, or motion-blurred edges—is a fundamental problem in computer vision known as matting. In the video domain, this task is further complicated by the requirement for temporal stability; inconsistent predictions lead to distracting flickering or "crawling" artifacts.

Traditional Video Object Matting (VOM) methods rely heavily on trimaps, which segment an image into foreground, background, and an "unknown" region where the alpha matte is calculated. Manually creating these for every frame is impractical. While automated trimap propagation tools exist, they often fail during fast motion or complex occlusions, leading to degraded results. The HSTSG framework addresses these issues by shifting from a trimap-centric pipeline to a unified space-time semantic guidance approach.

The HSTSG Framework

The core innovation of HSTSG lies in its ability to treat VOM as a space-time correspondence problem, similar to video object segmentation, but refined for the sub-pixel accuracy required for matting.

Hierarchical Memory Matching Module (HMMM)

Unlike standard propagation methods that operate at a single resolution, the HMMM extracts features at multiple scales (1/4, 1/8, and 1/16). A memory bank stores features from previous frames, allowing the model to find long-range correspondences. Coarse scales provide robust semantic localization, while finer scales preserve the intricate details necessary for high-quality alpha mattes. To maintain efficiency, a Top-k filtering strategy is employed, focusing computation only on the most relevant matching pixels in the memory bank.

Cross-Frame Attention Refinement (CFAR)

To ensure temporal coherence, the model utilizes a Transformer-based CFAR module. By analyzing a sliding window of adjacent frames, the module calculates correlations that "smooth out" predictions. This prevents the jittering common in frame-by-frame processing and ensures that the alpha matte evolves naturally over time, even during rapid movement.

Experimental Evaluation

The researchers evaluated HSTSG on two primary benchmarks: VideoMatting108, featuring high-resolution clips with complex foregrounds, and DVM, a large-scale dataset of real-world scenarios.

Quantitative Results

HSTSG demonstrated a significant leap in performance. On the DVM test set, it achieved a Mean Squared Error (MSE) of 5, a substantial improvement over the 14 recorded by the previous state-of-the-art (DVM). Crucially, HSTSG achieved this using only a single initial mask, whereas the baseline required full trimaps for every frame. Temporal stability metrics, such as dtSSD and MESSDdt, also showed marked improvements, confirming the effectiveness of the CFAR module.

Qualitative Comparison

As shown in the comparison below, traditional trimap propagation often results in noisy "unknown" regions, which in turn leads to blurred or artifact-heavy alpha mattes. HSTSG maintains a clean, sharp matte even as the object moves.

Comparison of HSTSG against traditional trimap propagation and TCVOM. Row (b) shows the failure of trimap propagation, leading to the artifacts in (c). The bottom row shows the superior results of the proposed HSTSG method.

Conclusion

HSTSG represents a significant advancement in user-friendly video editing. By replacing tedious trimap creation with a robust hierarchical propagation and refinement system, it enables high-quality matting with minimal manual effort. Future work may extend this spatio-temporal guidance to 3D reconstruction or explore "zero-shot" matting to further automate the object selection process.

Summary updated

BibTeX
@inproceedings{wang2023video,
  author = {Yumeng Wang and Bo Xu and Ziwen Li and Han Huang and Cheng Lu and Yandong Guo},
  title = {{Video Object Matting via Hierarchical Space-Time Semantic Guidance}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00509},
}
Figure