Lip language identification via Wavelet entropy and K-nearest neighbor algorithm

Ran Wang, Yifan Cui, Xinyu Gao, Wei Chen, Mingbo Hu, Qian Li, Jiahui Wei, Xianwei JiangEAI Endorsed Trans. e Learn 2021 · OpenReview

Abstract

Lip language identification, or lip-reading, remains a complex challenge in computer vision due to the ambiguous relationship between visual mouth shapes (visemes) and auditory sounds (phonemes). This paper proposes a streamlined identification system that leverages Wavelet Entropy (WE) for feature extraction and a K-Nearest Neighbor (KNN) algorithm for classification. By focusing on the "disorder" or complexity of lip movements rather than high-dimensional pixel data, the method achieves a peak accuracy of 80.08% on Mandarin Pinyin recognition. This approach offers a computationally efficient alternative to resource-intensive deep learning models, making it suitable for real-time applications on low-power devices.

Introduction

The primary difficulty in automated lip-reading lies in the "unstable" nature of lip shapes during speech. Multiple phonemes often produce nearly identical visemes, and traditional algorithms frequently struggle to capture the subtle transient characteristics of these movements. While modern deep learning approaches like Convolutional Neural Networks (CNNs) have advanced the state-of-the-art, their high computational overhead and requirement for massive datasets limit their deployment in constrained environments.

This research seeks a "middle ground" by combining elegant mathematical feature extraction with a simple, efficient classifier. The goal is to develop a system that is accurate enough for practical use while remaining lightweight and fast.

Methodology

The proposed pipeline consists of three primary stages: feature extraction via Wavelet Entropy, classification using K-Nearest Neighbor, and validation through K-fold cross-validation.

Wavelet Entropy (WE)

The core innovation of this work is the application of Wavelet Entropy to lip images. Instead of standard image descriptors, a wavelet transform is used to decompose lip images into various frequency scales. The Shannon entropy of these coefficients is then calculated to measure the energy distribution or "chaos" of the lip shape. Since lip movements are non-stationary signals, WE is uniquely suited to capturing dynamic characteristics while significantly reducing the feature space dimensionality.

Dataset and Classification

The study utilized a custom dataset of 520 lip images categorized into 26 classes based on Chinese Pinyin. These images capture the specific lifting, stretching, flattening, and turning of the lips required for Mandarin articulation.

Example of a rounded lip shape used in the Pinyin dataset.

For classification, the authors employed the K-Nearest Neighbor (KNN) algorithm. KNN classifies new samples by identifying the k most similar examples in the training set. Given the low-dimensional and highly descriptive nature of WE features, KNN provides an efficient classification mechanism without the training complexity of deep neural networks. To ensure robustness, 10-fold cross-validation was used to balance estimator bias and variance.

Results and Discussion

The WE+KNN method was evaluated against several established models. The system achieved an average accuracy of 78.30%, with a peak performance of 80.08%.

Model Accuracy
Proposed WE+KNN 80.08%
CD-DNN 77.19%
Eigen lips + SVM 70.60%
DNN-HMM 64.90%

Beyond accuracy, the proposed method demonstrated a significant reduction in calculation time and improved training speed compared to deep learning architectures. This efficiency highlights that for specific, constrained tasks like Pinyin recognition, classical signal processing combined with simple machine learning can outperform "brute force" deep learning models.

Conclusion

This paper demonstrates that treating lip movement as a signal-processing problem—specifically by measuring the entropy of wavelet coefficients—allows for highly effective dimensionality reduction. While the current dataset is relatively small and the KNN algorithm remains sensitive to outliers, the results are promising for real-time lip-reading applications. Future work will focus on expanding the dataset to include more complex speech patterns and testing the algorithm's robustness against environmental factors such as lighting variations and facial hair.

Summary updated

BibTeX
@article{wang2021lip,
  author = {Ran Wang and Yifan Cui and Xinyu Gao and Wei Chen and Mingbo Hu and Qian Li and Jiahui Wei and Xianwei Jiang},
  title = {{Lip language identification via Wavelet entropy and K-nearest neighbor algorithm}},
  journal = {EAI Endorsed Trans. e Learn.},
  year = {2021},
  url = {https://openreview.net/forum?id=0HECCuZJy3},
}
Figure