TL;DR
This paper introduces a novel method for automatically extracting key geometric lip features (lip aperture and width) from video using a combination of Discrete Cosine Transform (DCT) and Principal Component Analysis (PCA). The approach effectively compresses lip image information into a small set of predictors, achieving accurate estimations with significantly reduced dimensionality compared to traditional methods.
What Problem Does This Paper Address?
Visual speech processing, often referred to as lip-reading, plays a crucial role in enhancing speech perception, especially in noisy environments. While the benefits are well-established, automatically extracting meaningful visual features that correlate with phonetic information remains a challenge. Existing methods typically fall into two categories: shape-based features (relying on explicit lip contour extraction) and appearance-based features (using transformations like DCT or PCA on image pixels). This paper aims to bridge this gap by proposing a method that leverages the strengths of appearance-based features (DCT) to directly predict geometric shape features, offering a more efficient and potentially more robust approach. This is particularly relevant for projects like TELMA, which aims to develop automatic visual-to-speech translation systems for the hearing impaired.
Methods
The core of the proposed method involves a two-stage process:
- 2D Discrete Cosine Transform (DCT): The lip region of interest (ROI) is first extracted from the video frames. This ROI is then transformed using a 2D DCT. DCT is chosen for its energy compaction properties, meaning it concentrates the image's information into a few low-frequency coefficients, making it suitable for compression and feature extraction.
- Principal Component Analysis (PCA) and Multi-linear Modeling: A subset of the DCT coefficients is selected using a mask (initially a triangle or square shape). PCA is then applied to these selected coefficients to reduce dimensionality and decorrelate the features. This results in a set of uncorrelated principal components (factors). A multi-linear model is then built to predict the geometric lip features (lip width, aperture, and area) using a linear combination of these PCA factors. The model is optimized by minimizing the residual variance, effectively selecting the optimal mask size and shape, and the number of PCA factors needed for accurate prediction.
Datasets
The study utilized a custom dataset derived from a video recording of a female native French speaker. The dataset consists of:
- 50 isolated words, including 31 digits, 12 months, and 7 days.
- Video recorded at 50 frames per second (PAL format).
- Audio digitized at 22.050 kHz for temporal segmentation.
- Manually selected inner and outer lip contours converted to centimeters for ground truth geometric features (lip width, aperture, and area).
The dataset was split into a modeling set (811 frames) and a test set (759 frames).
Key Insights and Ideas
- Appearance to Shape Prediction: The paper demonstrates that appearance-based features (DCT coefficients) can be effectively used to directly predict geometric shape features, bypassing the need for explicit, potentially noisy, contour extraction.
- Dimensionality Reduction Power: By combining DCT with PCA and a mask, the method achieves significant dimensionality reduction. Instead of using all DCT coefficients, a small subset of PCA factors (around 19-21) can explain over 95% of the variance in lip features.
- Mask Optimization is Crucial: The choice of mask shape (triangle vs. square) and size significantly impacts the performance. The study found that a triangle mask of side 15 or a square mask of side 11 provided comparable good performance.
- Efficiency for Communication Technologies: The reduced set of predictors makes this method highly suitable for real-time applications and communication technologies for the hearing impaired, where computational efficiency is paramount.
- Robustness to Cued Speech Elements: Preliminary tests suggest that the presence of fingers near the lip ROI (as in Cued Speech) has a limited negative impact on the accuracy of the lip aperture estimation.
Key Results
- The method can estimate geometric lip features with a Root Mean Square error (RMSE) of 1 to 1.4 mm for lip aperture and lip width.
- For the lip aperture feature (B), approximately 19-21 PCA factors were sufficient to explain 95% of the variance in the modeling data and 93% in the test data. This is a significant reduction from the ~120 predictors that would be needed to gain only an additional 2% of explained variance.
- The model showed good performance on the test set, with residual variance decreasing rapidly and stabilizing after about 20 predictors.
- Preliminary tests indicated that the presence of fingers in the lip ROI did not significantly degrade the performance for lip aperture estimation (RMSE of 0.1406 cm with fingers vs. 0.1223 cm without).
Limitations and Future Work
- Weak Estimation Near Zero: The paper acknowledges a limitation where the model weakly estimates data points close to zero for certain features, as illustrated in Figure 6.
- Further Optimization: While the paper optimizes mask shape and size, further exploration of different mask types and more sophisticated feature selection techniques could be beneficial.
- Extension to Other Features: The current study focuses on lip aperture and width. Extending the model to predict other geometric features or even more complex phonetic features would be a valuable next step.
- Generalization: While the dataset is from a single speaker, testing the model's generalization capabilities across different speakers, accents, and speaking conditions is crucial for real-world applications.
- Integration with Audio: The paper mentions the TELMA project's goal of audio-visual speech translation. Future work could involve integrating this visual feature extraction with audio processing for a more robust speech recognition system.
Summary updated