Inner Lips Parameter Estimation based on Adaptive Ellipse Model

Li Liu, Gang Feng, Denis BeautempsAVSP 2017 · OpenReview

TL;DR

This paper introduces a novel, artifact-free method for estimating inner lip parameters (width and height) using an adaptive ellipse model. By combining image processing with an iterative ellipse fitting technique, the method achieves high accuracy in lip shape analysis, outperforming existing methods and showing promise for applications like Cued Speech recognition.

What Problem Does This Paper Address?

Accurately estimating the shape of the inner lips is crucial for various applications, including speech recognition, speech production analysis, and particularly for Cued Speech (CS), a communication method for the deaf that supplements lip-reading with hand gestures. However, lip tracking and parameter estimation are challenging due to the inherent variability of lip shapes, extreme movements, interference from teeth and tongue, inconsistent lighting, and potential occlusions (like hands in CS). Existing model-based approaches often require extensive training data or can be sensitive to these real-world variations. This paper aims to address this by developing a robust and efficient method for estimating inner lip width (A) and height (B) without relying on artificial markers or complex prior models.

Methods

The proposed method consists of two main stages:

  1. Image Pre-processing for Inner Lips Segmentation:
    • Initial Lips ROI Extraction: A Region of Interest (ROI) around the lips is first obtained using a pre-trained Constrained Local Neural Fields (CLNF) model, which is robust even with hand occlusions.
  2. Color-Based Segmentation: Within the lips ROI, a color-based approach is used to distinguish the inner lip area from teeth and darker regions. This involves analyzing luminance in the YCbCr color space and color ratios in the RGB space.
  3. Discontinuity Smoothing and Filling: To create a more complete and connected binary image of the inner lips, two techniques are employed:
  4. Single Discontinuity Smoothing: Eliminates isolated "black" pixels (non-lip areas) within the detected inner lip region.
  5. Horizontal/Vertical Block Filling: Fills small contiguous blocks of "black" pixels that might remain after the initial smoothing, ensuring a more solid inner lip contour representation.
  6. Adaptive Ellipse Model for Parameter Estimation:
    • Initial Ellipse Determination: An initial ellipse is determined by first finding a small rectangular region within the lips ROI that contains a significant portion of the white (inner lip) pixels. The center of this region is used as the initial ellipse center, and its initial semi-major and semi-minor axes are derived from the second moments of the white pixels within this region.
  7. Optimal Ellipse Searching: The initial ellipse then iteratively moves and expands in four directions (up, down, left, right). In each step, the ellipse's center and size are updated. The expansion stops when a criterion is met (e.g., the ratio of white pixels within the ellipse to the ellipse's total area falls below a threshold) or when the ellipse exceeds the lips ROI. This process converges to an optimal ellipse that best matches the segmented inner lip region.
  8. Parameter Extraction: The final semi-major axis ($\alpha$) and semi-minor axis ($b$) of the fitted ellipse are used to estimate the inner lip width (A) and height (B) using the formulas $A = \gamma \times 2\alpha$ and $B = \gamma \times 2b$, where $\gamma$ is a scaling factor determined experimentally.

Datasets

The study utilizes a custom dataset comprising 4693 images from three French speakers: one Cued Speech (CS) speaker and two male speakers. The videos were recorded at 576x720 resolution and 50fps. For evaluation, ground truth inner lip contours were manually extracted by an expert by placing landmarks. For the CS recognition experiment, 500 words (approximately 1500 phonemes) were used, with 80% for training and 20% for testing.

Key Insights and Ideas

  • Artifact-Free Estimation: The method's strength lies in its ability to estimate inner lip parameters without requiring artificial markers (like colored paint) or complex 3D models, making it more practical for real-world scenarios.
  • Robust Pre-processing: The combination of color-based segmentation and discontinuity filling techniques effectively handles variations in lighting and the presence of teeth/tongue, creating a reliable binary representation of the inner lip area.
  • Adaptive Ellipse Fitting: The iterative, adaptive ellipse fitting approach allows the model to conform to diverse lip shapes and movements, providing a good approximation of the inner lip contour.
  • Parameterization for CS: The estimated A and B parameters are shown to be valuable descriptors for Cued Speech recognition, demonstrating their utility beyond simple shape analysis.

Key Results

  • High Accuracy: The proposed method achieves an RMSE of 3.37 mm for the A parameter (inner lip width) and 0.84 mm for the B parameter (inner lip height) on the custom dataset.
  • Outperforms Baselines: This performance significantly surpasses existing state-of-the-art methods, including the Constrained Local Neural Fields (CLNF) approach, for inner lip parameter estimation.
  • Comparable to Raw ROI: In Cued Speech recognition tasks involving 34 French phonemes, the accuracy achieved using the estimated ellipse parameters was comparable to using raw lip ROIs, suggesting the parameters capture essential lip shape information.
  • Visual Validation: Figure 8 visually demonstrates the successful fitting of the adaptive ellipse to inner lips across different speakers and conditions, including hand occlusion in the CS case.

Limitations and Future Work

  • Subject and Lighting Dependence: The authors acknowledge that the image processing steps involve adjustable parameters (thresholds for dark area and teeth extraction) that are subject-dependent and sensitive to lighting variations. While they can be tuned experimentally, this dependence is a limitation.
  • Future Work - CNN Integration: The authors suggest that training a Convolutional Neural Network (CNN) on their database could help capture non-linear relationships in inner lip information, potentially reducing the dependence on manually tuned thresholds and improving robustness across different subjects.
  • Future Work - Speech Noise Reduction: The estimated lip parameters could also be applied to speech noise reduction techniques based on source separation.

Summary updated

Abstract

In this paper, a novel automatic method using an adaptive ellipse model to estimate inner lips parameters (inner lips width A and height B) of speakers without any artifices is presented. Color based image processing is first applied to delimit preliminary inner lips. A single discontinuity elimination combining horizontal and vertical filling are used to obtain a binary inner lips image as complete as possible. After the pre- processing steps, an optimal adaptive ellipse is determined to match the inner lips, giving A and B parameters. The proposed method is evaluated on 4693 images of three French speakers including one Cued Speech (CS) speaker. It obtains RMSE of 3.37 mm for A parameter and of 0.84 mm for B parameter which outperform the baseline of inner lips parameter estimation in the state of the art. Moreover, CS recognition based on 34 French phonemes shows that using the estimated two parameters achieves an accuracy which is comparable to that using raw lips ROI.
BibTeX
@inproceedings{liu2017inner,
  author = {Li Liu and Gang Feng and Denis Beautemps},
  title = {{Inner Lips Parameter Estimation based on Adaptive Ellipse Model}},
  booktitle = {AVSP},
  year = {2017},
  url = {https://openreview.net/forum?id=kZIBvwJ5VX},
}
Figure