Silent Speech Recognition with Strain Sensors and Deep Learning Analysis of Directional Facial Muscle Movement

Hyunjun Yoo, Eunji Kim, Jong Won Chung, Hyeon Cho, Sujin Jeong, Heeseung Kim, Dongju Jang, Hayun Kim, Jinsu Yoon, Gae Hwang Lee, Hyunbum Kang, Jooyoung Kim, Youngjun Yun, Sungroh Yoon, Yongtaek HongACS 2022 · DOI

Abstract

Silent communication based on biosignals from facial muscle requires accurate detection of its directional movement and thus optimally positioning minimum numbers of sensors for higher accuracy of speech recognition with a minimal person-to-person variation. So far, previous approaches based on electromyogram or pressure sensors are ineffective in detecting the directional movement of facial muscles. Therefore, in this study, high-performance strain sensors are used for separately detecting <i>x</i>- and <i>y</i>-axis strain. Directional strain distribution data of facial muscle is obtained by applying three-dimensional digital image correlation. Deep learning analysis is utilized for identifying optimal positions of directional strain sensors. The recognition system with four directional strain sensors conformably attached to the face shows silent vowel recognition with 85.24% accuracy and even 76.95% for completely nonobserved subjects. These results show that detection of the directional strain distribution at the optimal facial points will be the key enabling technology for highly accurate silent speech recognition.
BibTeX
@inproceedings{yoo2022silent,
  author = {Hyunjun Yoo and Eunji Kim and Jong Won Chung and Hyeon Cho and Sujin Jeong and Heeseung Kim and Dongju Jang and Hayun Kim and Jinsu Yoon and Gae Hwang Lee and Hyunbum Kang and Jooyoung Kim and Youngjun Yun and Sungroh Yoon and Yongtaek Hong},
  title = {{Silent Speech Recognition with Strain Sensors and Deep Learning Analysis of Directional Facial Muscle Movement}},
  booktitle = {ACS},
  year = {2022},
  doi = {10.1021/acsami.2c14918},
}

Loading related content...

Figure