Statistical estimators for use in automatic lip segmentation

Biswadev Goswami, William Christmas, Josef Kittler2006 · DOI

TL;DR

This paper investigates the use of statistical estimators, specifically the Minimum Covariance Determinant (MCD) estimator, for robustly segmenting lip regions in facial images. By comparing MCD with a non-robust estimator and a "hybrid edges" method, the authors demonstrate that robust statistical estimation offers a more reliable approach for lip segmentation, crucial for applications like speech processing and biometrics.

What Problem Does This Paper Address?

Accurately identifying and segmenting the lip region in facial images is a critical step for various applications, including speech processing (where lip movements can enhance speech recognition, especially in noisy environments) and biometric verification. While manual segmentation is feasible for a few images, it becomes a bottleneck when dealing with large datasets. Existing automatic methods often struggle with the inherent challenges of lip segmentation, such as variations in lighting, facial expressions, and the presence of "contextual outliers" like shadows, facial hair, or reflections. This paper aims to address the problem of reliably segmenting the lip region automatically by exploring the effectiveness of different statistical estimation techniques.

Methods

The core of the proposed system involves using statistical estimators to build a color model of the dominant spatial region in the lower half of a segmented face, which is assumed to be the "skin" region. This process then classifies pixels into "skin" and "not-skin" categories. The lip region is expected to fall within the "not-skin" class.

The paper compares two types of multi-variate statistical estimators:

  • Robust Multi-variate Statistical Estimator (MCD): The paper specifically uses the Minimum Covariance Determinant (MCD) estimator. MCD is designed to be robust to outliers by finding a subset of data points that has the minimum covariance determinant. This makes it less susceptible to contamination from artifacts like shadows or facial hair. The algorithm involves iterative steps of random sampling, calculating mean and covariance, and refining these estimates based on Mahalanobis distances.
  • Non-robust Multi-variate Statistical Estimator: This method uses the entire population of image pixels to estimate the mean and covariance, making it more vulnerable to outliers.

Following the statistical estimation and classification, a connected components analysis is performed on the "not-skin" region. A simple cost function is then implemented to identify the lip region. This cost function considers the Euclidean distance from the center of gravity of the input and the area of the labeled region, favoring larger regions closer to the center.

Datasets

The experiments were conducted using a sequence of 30 individual images from the XM2VTS database. This database is a well-known benchmark for face recognition and related tasks.

Key Insights and Ideas

  • Robustness is Key: The paper strongly advocates for the use of robust statistical estimators like MCD for lip segmentation. The presence of outliers (shadows, facial hair) significantly degrades the performance of non-robust methods.
  • Skin as the Dominant Region: The assumption that the "skin" region is the largest and most chromatically coherent area in the lower half of the face is a practical simplification that enables the application of statistical clustering.
  • Lip Region as "Not-Skin": By successfully segmenting the dominant "skin" region, the lip region is effectively isolated as part of the "not-skin" class, simplifying subsequent identification.
  • MCD's Advantage: The MCD estimator proved to be the most reliable method, particularly in cases with less specular reflection on the lips, which is a common challenge.
  • Hybrid Edges Limitations: The "hybrid edges" method, while providing reasonable results, is described as more heuristic and sensitive to dark outlier regions.

Key Results

The paper compares the performance of the MCD estimator, a non-robust estimator, and the "hybrid edges" method on 30 images from the XM2VTS database. The results are classified into "Correct," "Recoverable," and "Non-recoverable" categories.

  • MCD: Achieved 27 "Correct" segmentations, 2 "Recoverable," and 1 "Non-recoverable."
  • Non-robust: Achieved 26 "Correct" segmentations, 1 "Recoverable," and 3 "Non-recoverable."
  • Hybrid Edges: Achieved 21 "Correct" segmentations, 0 "Recoverable," and 9 "Non-recoverable."

In terms of speed, the non-robust estimator was ranked first, followed by the "hybrid edges" method, and then MCD. However, MCD's superior accuracy is highlighted as a significant advantage.

Limitations and Future Work

  • MCD Complexity: The primary disadvantage of the MCD method is its computational complexity. The paper uses a "Fast-MCD" implementation, but further improvements in computation time are desired, potentially through more efficient search and sort algorithms or alternative optimization methods.
  • Lip Region Modeling: The MCD algorithm implicitly assumes a single Gaussian distribution for the data it segments. However, the lip region is better modeled as a mixture of Gaussians, which the current MCD approach cannot directly segment.
  • Specular Reflections: The method struggles with specular reflections on the lips, and this issue, along with connected components analysis failures, will be investigated further.
  • Real-time Performance: Achieving real-time performance is a goal for future work.
  • Initialization and Tracking: The paper suggests using the non-robust estimation process to initialize the MCD algorithm or using a learned model to estimate lip clusters directly.
  • Morphological Operations: While morphological closing can be used to clean up results, it carries a risk of connecting the lip region to surrounding noise. The paper notes that closing can be applied after lip region identification to remove outliers.

Summary updated

Abstract

The past decade has seen a considerable increase in interest in the field of facial feature extraction. The primary reason for this is the variety of uses, in particular of the mouth region, in communicating important information about an individual which can in turn be used in a wide array of applications. The shape and dynamics of the mouth region convey the content of a communicated message, useful in applications involving speech processing as well as man-machine user interfaces. The mouth region can also be used as a parameter in a biometric verification system. Extraction of the mouth region from a face often uses lip contour processing to achieve these objectives. Thus, solving the problem of reliably segmenting the lip region given a talking face image is critical. This paper compares the use of statistical estimators, both robust and non-robust, when applied to the problem of automatic lip region segmentation. It then compares the results of the two systems with a state-of-the art method for lip segmentation.
BibTeX
@misc{goswami2006statistical,
  author = {Biswadev Goswami and William Christmas and Josef Kittler},
  title = {{Statistical estimators for use in automatic lip segmentation}},
  year = {2006},
  doi = {10.1049/cp:20061975},
}
Figure