Robust Portrait Image Matting and Depth-of-Field Synthesis via Multiplane Images

Zhefan Rao, Tianjia Zhang, Yuen Fui Lau, Qifeng ChenWACV 2025

TL;DR

This paper introduces a novel framework for generating realistic depth-of-field (DOF) effects and bokeh in portrait images captured by smartphones, using only a single all-in-focus image. By combining a multi-stage matting network for precise foreground extraction with a learning-free multiplane image (MPI) representation, the method achieves high-quality bokeh synthesis that rivals DSLR cameras.

What Problem Does This Paper Address?

Smartphones have largely replaced traditional cameras for everyday photography, but they often lack the ability to produce the natural depth-of-field blur (bokeh) characteristic of DSLR cameras. This is primarily due to limitations in lens aperture and sensor size. While smartphones offer "portrait modes" that simulate this effect, they often rely on simple blurring without considering true depth, leading to artifacts and unnatural results, especially around fine details like hair. Existing methods struggle with accurate foreground-background separation and generating realistic bokeh that varies with depth.

Methods

The proposed framework tackles this problem in three main stages:

  1. Multi-stage Matting Network: A novel multi-stage, multi-branch matting network is employed to generate a precise, strand-level portrait alpha mask. This network refines the mask iteratively, focusing on both global subject location and local details, which is crucial for handling intricate edges.
  2. Mask-based Depth Filtering: A pre-trained transformer-based depth estimation network provides an initial depth map. This depth map is then refined using a non-linear filter guided by the predicted alpha mask. This step is critical for aligning the depth information with the foreground mask, especially in transition areas, and correcting inaccuracies in the initial depth estimation.
  3. Learning-Free MPI Rendering: The filtered depth map and the refined alpha mask are used to construct a Multiplane Image (MPI) representation. Unlike previous methods that learn MPIs, this approach uses a learning-free optimization mechanism. The MPI consists of multiple layers, each representing a plane at a specific depth. Each layer contains a blurred image and an alpha channel. The blur kernel size is proportional to the distance from the focus plane, and the alpha channel controls visibility. This MPI is then used to render the final image with depth-aware bokeh effects through an over-composition process from back to front.

Datasets

  • Matting Task: P3M-10k dataset, comprising 9,421 blurred and 500 clear portrait images for training and validation.
  • Bokeh Effect Rendering Task:
    • WAX dataset: 20 photos of wax figures with different backgrounds, used for quantitative optimization and evaluation.
  • REAL dataset: Approximately 100 real-person images captured by smartphones, intended to better match the expected input data distribution.

Key Insights and Ideas

  • The importance of precise matting for bokeh: The paper highlights that accurate foreground-background separation is not just for aesthetics but is fundamental for correctly applying depth-aware blur. Inaccurate mattes lead to artifacts where the background bleeds into the foreground or vice-versa.
  • Learning-free MPI optimization: By avoiding a neural network for MPI generation, the method becomes more flexible, requires less data, and allows for easier parameter tuning, making it more practical for real-world applications.
  • Depth-guided depth map refinement: Directly using a depth map for rendering can be problematic due to inconsistencies. Aligning it with a high-quality alpha mask significantly improves the realism of the synthesized DOF effects.
  • Transforming to an exponential space for contrast: Simple convolution with disk kernels can reduce image contrast. Transforming the image to an exponential space before blurring and then back helps preserve and enhance contrast, leading to more salient bokeh balls.

Key Results

  • Matting Performance: The proposed matting pipeline achieves state-of-the-art results on the P3M-10k dataset, demonstrating superior accuracy, particularly in challenging hair areas, compared to methods like MODNet and P3M. Quantitative metrics (MAD↓, MSE↓) show improvements in both blurry and non-blurry scenarios.
  • Bokeh Rendering Quality: Qualitative results on the REAL and WAX datasets show that the method produces more natural and realistic bokeh effects than existing methods like BokehMe and MPIB. It effectively eliminates halos at object boundaries and creates a more convincing depth gradient.
  • Quantitative Rendering Metrics: On the WAX dataset, the proposed method achieves competitive or superior results in PSNR and SSIM compared to baseline methods, while also achieving a lower LPIPS score, indicating better perceptual similarity.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that convolutions involving large kernels can be computationally intensive, posing a limitation for real-time processing and ultra-high-definition rendering.
  • Future Directions: The authors suggest optimizing convolution algorithms or exploring hardware acceleration to address the computational cost. This would enhance the framework's applicability in demanding scenarios and improve its overall usability for real-time image processing and rendering.

Summary updated

BibTeX
@inproceedings{rao2025robust,
  author = {Zhefan Rao and Tianjia Zhang and Yuen Fui Lau and Qifeng Chen},
  title = {{Robust Portrait Image Matting and Depth-of-Field Synthesis via Multiplane Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure