High-Resolution Deep Image Matting

Haichao Yu, Ning Xu, Zilong Huang, Yuqian Zhou, Humphrey Shi

Abstract

Image matting is a fundamental technique in image and video editing, yet traditional deep learning approaches often struggle with high-resolution (HR) inputs due to hardware memory constraints. This paper introduces HDMatt, the first deep learning-based framework specifically designed for high-resolution image matting. By employing a novel "crop-and-stitch" strategy integrated with a Cross-Patch Contextual (CPC) module, HDMatt effectively captures both fine-grained local details and global contextual dependencies. Experimental results on the Adobe Image Matting and AlphaMatting benchmarks demonstrate that HDMatt achieves state-of-the-art performance, particularly on real-world ultra-high-resolution images.

The Problem: The "Memory Wall" in Matting

Precise foreground extraction, or alpha matting, is critical for high-quality composition. While deep learning has significantly improved matting accuracy, modern high-resolution images (e.g., 6000x6000px) exceed the memory capacity of standard GPUs. Researchers typically resort to two suboptimal workarounds:

  1. Downsampling: Reducing image size to fit memory, which destroys the fine details (hair, fur, translucent edges) that matting is intended to preserve.
  2. Naive Cropping: Processing the image in independent patches. This leads to a loss of global context; without seeing the broader scene, a model may fail to distinguish between similar foreground and background textures, resulting in inconsistent artifacts across patch boundaries.

Methodology: HDMatt and the CPC Module

HDMatt addresses these issues by processing images in a patch-based manner while maintaining global awareness through a specialized architecture.

Encoder-Decoder with ASPP

The core architecture utilizes a ResNet-34 backbone. It incorporates Atrous Spatial Pyramid Pooling (ASPP) to extract multi-scale features, ensuring the model can handle objects of varying sizes within patches.

Cross-Patch Contextual (CPC) Module

The CPC module is the primary innovation of HDMatt. Instead of treating each patch in isolation, the CPC module allows a "query patch" to sample and attend to "reference patches" from across the entire image. By calculating similarity scores, the model identifies the most relevant context to inform the matting process in the current patch.

Trimap-Guided Non-Local (TGNL) Operation

To further refine the context, HDMatt uses the provided trimap (which labels regions as foreground, background, or unknown). The TGNL operation explicitly propagates information from known foreground and background regions in reference patches to the unknown regions of the query patch, ensuring structural and color consistency.

Evaluation and Results

HDMatt was evaluated using the Adobe Image Matting (AIM) dataset, AlphaMatting.com, and a new HR Real-World Dataset featuring images up to 6000x6000px.

Qualitative comparison of HDMatt against state-of-the-art methods on a high-resolution image.

Key Findings

  • State-of-the-Art Accuracy: HDMatt achieved the top rank on the AlphaMatting.com benchmark at the time of submission, leading in SAD (Sum of Absolute Differences), MSE, and Gradient metrics.
  • Quantitative Performance: On the AIM test set, HDMatt achieved a SAD of 33.5, outperforming previous leaders like ContextNet (35.8).
  • Visual Fidelity: As shown in the comparative results, HDMatt preserves intricate details that are often blurred by downsampling-based methods or rendered inconsistently by naive patch-based approaches.

Conclusion and Future Work

HDMatt successfully bridges the gap between deep learning matting models and high-resolution real-world requirements. While the model sets a new standard, the authors note that performance is still limited by the resolution and detail of existing training datasets like AIM. Future research will focus on improving inference speed—as patch-based processing is computationally intensive—and developing more diverse, ultra-high-resolution training sets to further refine the model's ability to capture the finest possible details.

Summary updated

Abstract

Image matting is a key technique for image and video editing and composition. Conventionally, deep learning approaches take the whole input image and an associated trimap to infer the alpha matte using convolutional neural networks. Such approaches set state-of-the-arts in image matting; however, they may fail in real-world matting applications due to hardware limitations, since real-world input images for matting are mostly of very high resolution. In this paper, we propose HDMatt, a first deep learning based image matting approach for high-resolution inputs. More concretely, HDMatt runs matting in a patch-based crop-and-stitch manner for high-resolution inputs with a novel module design to address the contextual dependency and consistency issues between different patches. Compared with vanilla patch-based inference which computes each patch independently, we explicitly model the cross-patch contextual dependency with a newly-proposed Cross-Patch Contextual module (CPC) guided by the given trimap. Extensive experiments demonstrate the effectiveness of the proposed method and its necessity for high-resolution inputs. Our HDMatt approach also sets new state-of-the-art performance on Adobe Image Matting and AlphaMatting benchmarks and produce impressive visual results on more real-world high-resolution images.
BibTeX
@inproceedings{yu2021high,
  author = {Haichao Yu and Ning Xu and Zilong Huang and Yuqian Zhou and Humphrey Shi},
  title = {{High-Resolution Deep Image Matting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=8Sl4OvkBIUj},
}
Figure