Abstract
The goal of natural image matting is the estimation of opacities of a user-defined foreground object that is essential in creating realistic composite imagery. Natural matting is a challenging process due to the high number of unknowns in the mathematical modeling of the problem, namely the opacities as well as the foreground and background layer colors, while the original image serves as the single observation. In this paper, we propose the estimation of the layer colors through the use of deep neural networks prior to the opacity estimation. The layer color estimation is a better match for the capabilities of neural networks, and the availability of these colors substantially increase the performance of opacity estimation due to the reduced number of unknowns in the compositing equation. A prominent approach to matting in parallel to ours is called sampling-based matting, which involves gathering color samples from known-opacity regions to predict the layer colors. Our approach outperforms not only the previous hand-crafted sampling algorithms, but also current data-driven methods. We hence classify our method as a hybrid sampling- and learning-based approach to matting, and demonstrate the effectiveness of our approach through detailed ablation studies using alternative network architectures.
Introduction
Natural image matting is the process of extracting a foreground object from an image with soft transitions, such as hair, fur, or glass. Mathematically, this process is governed by the compositing equation:
$$I_i = \alpha_i F_i + (1 - \alpha_i) B_i$$
where $I$ is the observed pixel color, and $\alpha$ (opacity), $F$ (foreground color), and $B$ (background color) are all unknowns. This problem is severely under-constrained; for every pixel, there is only one observation but seven unknowns (three for $F$, three for $B$, and one for $\alpha$).
Existing methods generally fall into two categories: deep learning-based methods that attempt to predict $\alpha$ directly, and traditional sampling-based methods that try to estimate $F$ and $B$ by looking at nearby known pixels. While deep learning has shown promise, it often struggles with fine details. Conversely, traditional sampling methods often fail to capture the global structure or texture of a scene. This paper proposes a hybrid approach that leverages the strengths of deep learning to perform "sampling" by predicting the underlying layer colors.
Methods
The proposed method decomposes the matting problem into a three-stage pipeline that treats layer color estimation as a guided scene completion or inpainting task. By estimating $F$ and $B$ first, the subsequent matting network has a significantly reduced search space for calculating the final opacity.
The Three-Stage Pipeline
- Background Sampling ($S_B$): A deep network estimates the background color ($B$) for the unknown regions. Unlike standard inpainting, this network uses the original image as a guide, as the background is often partially visible through the foreground object.
- Foreground Sampling ($S_F$): A second network estimates the foreground color ($F$). It takes the original image, the trimap, and the previously estimated background as inputs to hallucinate the foreground texture behind the background.
- Matting Network (MAT): Finally, a generative network takes the original image, the trimap, and the predicted $F$ and $B$ layers to produce the final alpha matte ($\alpha$).
Training and Innovations
To ensure high-quality results, the authors introduced a Gradient Loss ($L_{grad}$). This loss function is based on the spatial gradient of the alpha matte, encouraging the network to produce sharp, high-frequency details rather than the blurry edges common in many deep learning approaches. Additionally, the model utilizes Dynamic Data Augmentation, creating "hybrid scenes" during training by compositing different foreground objects and randomly jittering colors, scales, and trimap dilations to improve generalization.
Key Results
The effectiveness of this hybrid approach was demonstrated on the Alpha Matting benchmark, where the method ranked #1 at the time of publication. It outperformed both pure deep learning models, such as Deep Matting, and traditional sampling methods.
Quantitative ablation studies confirmed that adding background and foreground color predictions significantly reduced the Mean Squared Error (MSE) and Sum of Absolute Differences (SAD) compared to models that attempted to predict $\alpha$ directly. Visually, the method excels at preserving fine structures, such as the thin leaves of a plant or individual strands of hair, which are often lost or blurred by competing algorithms. Despite the multi-stage architecture, the system remains efficient, processing benchmark images in approximately 0.6 seconds.
Conclusion
By reframing traditional sampling as a learning-based inpainting task, this research demonstrates that decomposing the matting problem into sub-tasks is highly effective. Estimating layer colors ($F$ and $B$) is a more convenient task for convolutional neural networks than direct opacity estimation. While the model is subject to potential error propagation from the sampling stages, its state-of-the-art performance suggests that guided color estimation is a powerful tool for solving under-constrained computer vision problems.
Summary updated