Not Only Vision: Evolve Visual Speech Recognition via Peripheral Information

Zhaoxin Yuan, Shuang Yang, Shiguang Shan, Xilin ChenSubmission to ICLR 2025 · Withdrawn

Abstract

Visual Speech Recognition (VSR) aims to infer what was said by analyzing the speaker's facial dynamics. However, is reliance solely on visual information sufficient in challenging real-world scenarios? In human visual perception, peripheral vision refers to non-central areas of the visual field, crucial for providing overall awareness and detailed perception of central objects. Similarly, human lip-readers do not rely exclusively on lip movements but integrate contextual cues and prior knowledge to achieve more accurate transcribing. For the first time in machine lip-reading, we frame these non-lip-movement factors into a new concept of semantic-level peripheral information, Specifically, we select three representative types varying in relevance to the spoken content: (1) Contextual peripheral information, such as the general topic or some basic knowledge of the speech, can significantly narrow the range of potential recognition hypotheses. (2) Experiential peripheral information emerges from the recognition process itself. The very act of recognizing speech in a specific language provides implicit knowledge of grammar, word collocations, and related linguistic aspects, thereby guiding the recognition effectively. (3) Perturbative peripheral information introduces disturbance factors into the recognition process, analogous to noise injection in visual tasks. Semantic-level peripheral information is indirectly linked to transcripts; thus fusing it into VSR necessitates strong contextual understanding and inference capabilities. Here, we propose a multimodal learning framework built on a large language model (LLM), leveraging its powerful contextual modeling capabilities to take advantage of peripheral information. Our method's efficacy is demonstrated on two popular datasets. On the widely-used LRS3 dataset, we achieved a Word Error Rate (WER) of 24.5\% with readily available peripheral information, leading to an impressive 14.3\% relative improvement over the model without such information. To the best of our knowledge, our work sets a new state-of-the-art when utilizing similar hours of lip-reading videos. We further reported the evaluation on the more challenging AVSpeech dataset. Results across both datasets and various experimental settings demonstrate the promising potential of the proposed semantic-level peripheral information for VSR.
BibTeX
@misc{yuan2025not,
  author = {Zhaoxin Yuan and Shuang Yang and Shiguang Shan and Xilin Chen},
  title = {{Not Only Vision: Evolve Visual Speech Recognition via Peripheral Information}},
  year = {2025},
  url = {https://openreview.net/forum?id=TykT5YB89r},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure