High-Resolution Deep Image Matting

Haichao Yu, Ning Xu, Zilong Huang, Yuqian Zhou, Humphrey Shi

Abstract

Image matting is a key technique for image and video editing and composition. Conventionally, deep learning approaches take the whole input image and an associated trimap to infer the alpha matte using convolutional neural networks. Such approaches set state-of-the-arts in image matting; however, they may fail in real-world matting applications due to hardware limitations, since real-world input images for matting are mostly of very high resolution. In this paper, we propose HDMatt, a first deep learning based image matting approach for high-resolution inputs. More concretely, HDMatt runs matting in a patch-based crop-and-stitch manner for high-resolution inputs with a novel module design to address the contextual dependency and consistency issues between different patches. Compared with vanilla patch-based inference which computes each patch independently, we explicitly model the cross-patch contextual dependency with a newly-proposed Cross-Patch Contextual module (CPC) guided by the given trimap. Extensive experiments demonstrate the effectiveness of the proposed method and its necessity for high-resolution inputs. Our HDMatt approach also sets new state-of-the-art performance on Adobe Image Matting and AlphaMatting benchmarks and produce impressive visual results on more real-world high-resolution images.

Introduction

Image matting—the process of precisely separating a foreground object from its background by estimating an "alpha matte"—is essential for high-quality photo editing. While deep learning has pushed the state-of-the-art in this field, most models struggle with real-world high-resolution (HR) images, such as 4K or 8K.

An example of a high-resolution alpha matte generated by HDMatt, capturing fine details of a dandelion-like seed.

The primary bottleneck is hardware: feeding a 5000x5000 image into a standard deep neural network exceeds the memory capacity of even high-end GPUs. Researchers typically resort to downsampling, which destroys fine details like individual hairs, or trivial cropping, which leads to patch-wise inconsistency because the model lacks global context to resolve background ambiguity.

Methodology: The HDMatt Approach

The authors propose HDMatt, which processes images in patches but ensures those patches "talk" to each other to maintain global consistency.

Patch-Based Inference

The image is divided into overlapping patches. After processing, a specialized blending function is used to stitch them back together, ensuring smooth transitions and eliminating visible seams. Ablation studies showed that the architecture is robust and works well regardless of the specific patch size used, making it flexible for various hardware constraints.

Cross-Patch Contextual (CPC) Module

This module is the core innovation. For any given "query patch," the CPC module identifies and samples the top-K most relevant "context patches" from elsewhere in the image. This allows the model to understand the global environment (e.g., what the background looks like in a clear area) even when looking at a zoomed-in patch of complex foreground.

Trimap-Guided Non-Local (TGNL) Operation

Within the CPC module, the model uses the user-provided trimap (labeling regions as foreground, background, or unknown) to guide information flow. It explicitly compares the "unknown" pixels in the current patch against the known foreground and background pixels in context patches to make a more informed prediction.

Experiments and Key Results

The model was evaluated using standard benchmarks like Adobe Image Matting (AIM) and AlphaMatting.com, as well as a new HR real-world dataset collected by the authors featuring images up to 6000x6000 pixels.

Qualitative comparison of HDMatt. (a) HR Input Image, (b) Trimap, (c) Ground Truth, (d-f) Comparison of different matting results.

HDMatt achieved the #1 spot on the AlphaMatting.com benchmark at the time of submission and outperformed existing models like ContextNet and IndexNet on the AIM dataset across all metrics (SAD, MSE, Gradient, and Connectivity). On 6K images, HDMatt produced sharp, consistent results, while downsampling methods produced blurry edges and naive cropping methods produced visible "blocky" artifacts.

Conclusion

HDMatt demonstrates that high-resolution matting requires a balance between local detail and global context. By sampling just a few distant patches, the model gains enough context to stay consistent without the computational overhead of processing the entire high-res image at once. While the model is highly effective, future work may focus on capturing even finer textures and extending the cross-patch dependency logic to video matting for temporal consistency.

Summary updated

Abstract

Image matting is a key technique for image and video editing and composition. Conventionally, deep learning approaches take the whole input image and an associated trimap to infer the alpha matte using convolutional neural networks. Such approaches set state-of-the-arts in image matting; however, they may fail in real-world matting applications due to hardware limitations, since real-world input images for matting are mostly of very high resolution. In this paper, we propose HDMatt, a first deep learning based image matting approach for high-resolution inputs. More concretely, HDMatt runs matting in a patch-based crop-and-stitch manner for high-resolution inputs with a novel module design to address the contextual dependency and consistency issues between different patches. Compared with vanilla patch-based inference which computes each patch independently, we explicitly model the cross-patch contextual dependency with a newly-proposed Cross-Patch Contextual module (CPC) guided by the given trimap. Extensive experiments demonstrate the effectiveness of the proposed method and its necessity for high-resolution inputs. Our HDMatt approach also sets new state-of-the-art performance on Adobe Image Matting and AlphaMatting benchmarks and produce impressive visual results on more real-world high-resolution images.
BibTeX
@inproceedings{yu2021high,
  author = {Haichao Yu and Ning Xu and Zilong Huang and Yuqian Zhou and Humphrey Shi},
  title = {{High-Resolution Deep Image Matting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=8Sl4OvkBIUj},
}
Figure