TL;DR
This paper tackles the challenge of automatically extracting lip contours from un-marked, front-facing videos for visual speech processing. The authors adapt a state-of-the-art face landmark detection model (CLNF) and propose novel post-processing techniques to improve its accuracy, particularly for challenging cases like lower lip misdetections and small lip openings.
What Problem Does This Paper Address?
Accurate lip contour extraction is crucial for visual speech processing, enabling applications like lip-reading for the hearing impaired and enhancing speech recognition in noisy environments. Historically, this has often relied on artificial markers (like blue makeup) to simplify contour detection. However, this is impractical for real-world scenarios. The paper aims to overcome this by developing a method that works on "natural" (unmarked) faces, specifically addressing the limitations of existing automatic methods, like the CLNF model, which, while promising, still struggle with certain lip configurations and can produce errors.
Methods
The core of the proposed method is the Constrained Local Neural Field (CLNF) model, a computer vision algorithm designed for robust facial landmark detection. CLNF, trained on a large dataset of faces, provides 8 characteristic points that define the inner lip contour.
However, CLNF's direct application to the authors' audio-visual data yielded satisfactory results in only about 70% of cases, with specific issues arising for the lower lip and small lip openings. To address these shortcomings, the authors introduce two key post-processing strategies:
- Lower Lip Correction: For cases where CLNF misidentifies the lower lip, a method based on detecting the maximum luminance gradient in the vertical direction is employed. This approach leverages the sharp change in brightness between the teeth/tongue and the lower lip. A spline smoothing technique is used to make this gradient calculation robust to noise. The need for this correction is identified by a high A/B ratio (lip parameters), which is a stable indicator of CLNF errors.
- Small Lip Opening Correction: For rounded lips with small openings, CLNF's predicted outer corner points are often inaccurate. Instead of relying on these points, the authors propose an interpolation method that uses the 6 central lip points provided by CLNF. This involves scaling the vertical coordinates to form a square, converting to polar coordinates for smooth interpolation, and then converting back. This method is specifically applied when the A/B ratio exceeds a certain threshold, indicating this particular type of error.
Datasets
The study utilizes a custom dataset of 179 images. These images are derived from video recordings of vowel sounds from a corpus of 50 isolated French words, spoken by a single subject. The dataset was manually annotated by an expert who placed approximately forty points to define the inner lip contour.
Key Insights and Ideas
- CLNF's Promise and Pitfalls: CLNF is a powerful generic model for facial landmark detection, but its direct application to specific tasks like lip contour extraction requires fine-tuning and error correction.
- Luminance Gradient as a Lip Detector: The sharp contrast in luminance at the boundary of the lower lip provides a reliable cue for correction, even in the presence of noise.
- Parameter Ratios as Error Indicators: The ratio of lip parameters A (stretching) and B (aperture) serves as an effective heuristic to detect specific types of CLNF errors, guiding the application of correction methods.
- Interpolation Strategies Matter: Simple linear interpolation is insufficient for complex lip contours. Spline interpolation for the lower lip and a specialized polar coordinate interpolation for small openings are crucial for achieving accurate results.
- Unmarked Lips are Achievable: The paper demonstrates that accurate lip contour extraction is feasible without artificial markers, opening doors for more naturalistic applications.
Key Results
- The CLNF model directly achieves satisfactory results in approximately 70% of cases.
- The proposed correction methods significantly improve accuracy. For the lower lip correction, the mean quadratic error for parameter B is reduced from 0.3 cm to 0.1 cm.
- For small lip openings, the proposed interpolation method reduces the mean quadratic error for parameter A from 1.0 cm to 0.4 cm.
- The mean quadratic error for parameter S (area) is reduced from 0.89 cm² to 0.35 cm².
- The final results achieve accuracies comparable to literature values, but notably, without the use of lip makeup.
Limitations and Future Work
- Limited Corpus: The current dataset consists of only 179 images from a single speaker. Future work should involve expanding the corpus to include a wider variety of speech units and multiple speakers to ensure generalizability.
- Specific Error Types: While the proposed methods address key CLNF errors, they are tailored to specific issues. Further research could explore more comprehensive error handling.
- Complex Scenarios: The method is primarily tested on front-facing, clear images. Future work should investigate its performance in more complex situations, such as occlusions (e.g., hand-to-face) and variations in recording conditions.
- Core Model Limitations: The proposed improvements do not alter the core CLNF model itself. Further advancements might involve modifying the CLNF architecture or training data.
Summary updated