A Survey on Natural Image Matting With Closed-Form Solutions

Xiaoqiang Li (李晓强), Xiaoqiang Li, Jide Li, Hong Lu

Abstract

This paper presents a theoretical unification of "closed-form" image matting techniques—methods that solve for foreground opacity by minimizing a quadratic cost function. The authors propose a three-level hierarchical model to classify existing affinity-based methods and utilize manifold learning theory to explain how these algorithms map complex image colors to a simple 1-D alpha matte.

Introduction

Image matting is the process of separating a foreground object from its background by estimating an "alpha matte" ($\alpha$), where each pixel value ranges from 0 (pure background) to 1 (pure foreground). This is a notoriously "ill-posed" problem: for every pixel in a standard RGB image, we have three known values but seven unknowns (the RGB of the foreground, the RGB of the background, and the alpha value).

While many algorithms exist, they are often categorized haphazardly as "affinity-based" (looking at neighboring pixels) or "sampling-based" (searching the image for similar colors). The authors identify a lack of systematic theoretical structure in the field and aim to bridge this gap by providing a unified mathematical framework.

The Image Matting Problem

The fundamental goal of image matting is to solve the compositing equation: $$I = \alpha F + (1 - \alpha)B$$ where $I$ is the observed image, $F$ is the foreground, $B$ is the background, and $\alpha$ is the opacity.

The image matting process: decomposing an input image $I$ using a trimap $T$ into an alpha matte $\alpha$, foreground $F$, and background $B$.

Hierarchical Matting Model

The core of the paper is the introduction of a Hierarchical Matting Model that categorizes affinity-based approaches into three levels of increasing complexity:

  1. Level 1: Color Smooth Model: Assumes that if colors are similar, their alpha values should be similar. This is the foundation for methods like Random Walk matting.
  2. Level 2: Alpha-Color Model: Assumes a linear relationship between a pixel's color and its alpha value within a small window. This is the basis for the famous "Closed-Form Matting" (Levin et al.).
  3. Level 3: Color Mixture Model: Assumes a pixel's color (and its alpha) can be represented as a linear combination of its neighbors. This is used in Learning-Based (LB) matting.

Manifold Learning Perspective

The authors demonstrate that all three levels can be unified under Manifold Learning Theory. They frame matting as an operation that maps a high-dimensional image color manifold onto a low-dimensional (1-D) interval $[0, 1]$. By deriving the "Matting Laplacian" matrix ($L$) for each level, they show that the optimal alpha matte can always be found by solving a sparse linear system of the form $(L + \lambda C)\alpha = \lambda C \alpha^*$.

Key Insights and Results

The authors evaluate representative algorithms using standard benchmarks like alphamatting.com and perceptually motivated video datasets.

  • The "Closed-Form" Umbrella: The authors argue that "closed-form" should refer to any method that derives a quadratic cost function solvable via linear equations, unifying a wide range of techniques.
  • Sampling as a Bridge: Local affinity methods often fail when a foreground region is "isolated." Sampling acts as a "non-local affinity," building shortcuts across the manifold to propagate labels to these regions.
  • Deep Learning Dominance: The Deep Image Matting (DI) algorithm generally outperforms traditional methods in terms of SAD and MSE. However, traditional closed-form solutions provide essential "smoothness priors" that can be integrated into neural networks to improve boundary precision.
  • Traditional Top Performers: Among traditional methods, Information Flow (IF) and LNSP (which combines local and non-local priors) achieve the lowest error rates on static images.

Limitations and Future Work

Despite the theoretical unification, several hurdles remain:

  • The Semantic Gap: Traditional solutions rely purely on low-level color statistics and cannot distinguish between semantically different objects with similar colors. Future work must better integrate high-level semantic features (e.g., via CNNs).
  • Computational Efficiency: High-precision matting is computationally expensive, posing a bottleneck for real-time video applications.
  • User Burden: Most methods still require a "trimap" (a hand-drawn map of known regions). The authors suggest moving toward more "goal-oriented" interactive tools that guide the user to areas of high algorithmic uncertainty.

Summary updated

Abstract

Natural image matting refers to the interactive process of estimating the foreground opacity (alpha matte) of an input image and extracting the foreground layer. Various approaches have been proposed utilizing affinity-based strategy, sampling-based strategy or hybrid of both. Affinity-based approaches and hybrid approaches generally derive a quadratic cost function concerning alpha matte smoothness as well as user input and estimate the alpha matte through solving a sparse linear system of equations. These approaches, which have closed-form solutions, are called closed-form matting approaches. This paper provides an extensive overview on the closed-form matting theory. A hierarchical matting model is presented, classifying affinity-based approaches into three categories. The basis of each model and the relationships between them are analyzed. Using manifold learning theory, we unify the closed-form matting approaches into one framework and provide explanations of: 1) how various affinity-based strategies generate the alpha matte; 2) why sampling-based strategies help improve the precision of alpha matte. Evaluation of the representative algorithms on an image matting dataset and a video matting dataset demonstrates the strength and weakness of each matting model. Not only summary of the traditional matting methods is provided but also several suggestions on future directions for matting research are proposed.
BibTeX
@article{2019survey,
  author = {Xiaoqiang Li (李晓强) and Xiaoqiang Li and Jide Li and Hong Lu},
  title = {{A Survey on Natural Image Matting With Closed-Form Solutions}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2941832},
}
Figure