Image matting is the process of precisely extracting a foreground object from an image by estimating its per-pixel opacity, known as an alpha matte ($\alpha$). Unlike binary segmentation, which simply decides whether a pixel is "in" or "out," matting accounts for the soft transitions found in hair, fur, smoke, or motion blur.
The Composition Equation
At the heart of image matting is a simple linear blending model. For any observed pixel color $I$, we assume it is a combination of a foreground color $F$ and a background color $B$, weighted by the opacity $\alpha$:
$$I = \alpha F + (1 - \alpha) B$$
When $\alpha = 1$, the pixel is purely foreground; when $\alpha = 0$, it is purely background. The challenge lies in the "mixed" pixels where $0 < \alpha < 1$. As noted in A Closed Form Solution to Natural Image Matting, this is a "massively ill-posed" problem. For every single pixel, we observe only three values (the Red, Green, and Blue channels of $I$), but we must solve for seven unknowns: the RGB values of $F$, the RGB values of $B$, and the scalar $\alpha$.
Because there are more unknowns than equations, an infinite number of combinations of $F$, $B$, and $\alpha$ can produce the same observed color $I$. To resolve this ambiguity, researchers must introduce external constraints or "priors."
Classical Priors and the Matting Laplacian
Early breakthroughs in matting relied on the assumption of local smoothness. Anat Levin and colleagues proposed that within a small window, foreground and background colors are likely to be roughly constant or follow a linear trend.
This led to the derivation of the Matting Laplacian, a mathematical operator that describes the relationships between neighboring pixels. By minimizing a cost function based on this Laplacian, researchers could solve for the alpha matte using only a few user-provided scribbles. This approach, further refined by Dongqing Zou in Image Matting with Local and Nonlocal Smooth Priors, allowed for high-quality extraction without the need for a pixel-perfect "trimap" (a hand-drawn map labeling regions as definitely foreground, definitely background, or unknown).
Learning the Priors
With the advent of deep learning, the field shifted from hand-crafted mathematical priors to learned ones. Ning Xu introduced the first end-to-end deep learning approach in Deep Image Matting, training a network on a massive dataset to recognize what "natural" foregrounds and backgrounds look like.
Modern architectures have become increasingly sophisticated in how they handle guidance:
- Semantic Guidance: Yanan Sun proposed Semantic Image Matting, which teaches models to recognize specific materials (like hair vs. mesh) to apply different matting strategies.
- Transformer-based Context: GyuTae Park developed MatteFormer, using self-attention to allow local patches to "borrow" information from distant, similar-looking regions of the image.
- Interactive Refinement: To reduce the burden on users, Tianyi Wei and Xin Yang developed systems that use uncertainty estimation to ask for user clicks only in the most ambiguous areas.
Generative Matting and Diffusion
The most recent frontier treats matting not as a simple regression task, but as a generative one. Shengfeng He and others have introduced DiffusionMat, which uses Latent Diffusion Models to iteratively refine a noisy matte into a high-fidelity result.
By leveraging the vast "knowledge" of natural images stored in pre-trained diffusion models, frameworks like DRIP can jointly predict both the alpha matte and the true foreground color. This generative approach effectively "imagines" the hidden parts of the foreground that are partially obscured by the background, solving the ill-posed problem by picking the most statistically likely solution from a learned distribution of natural images.