OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment

Xize Cheng, Tao Jin, Linjun Li, Wang Lin, Xinyu Duan, Zhou Zhao

Abstract

Training effective Visual Speech Recognition (VSR) models is hindered by the scarcity of high-quality labeled video data. We present OpenSR, a novel training framework that enables zero-shot lip-reading by leveraging abundant labeled audio data. By maintaining a strictly aligned multi-modal phoneme space through a frozen encoder, OpenSR allows a decoder trained exclusively on audio to generalize to visual-only and audio-visual tasks. OpenSR achieves state-of-the-art results on the LRS2 benchmark, with its zero-shot lip-reading performance surpassing several models trained on hundreds of hours of labeled video.

Introduction

Visual Speech Recognition (VSR), or lip-reading, faces a significant "data hunger" problem. While labeled audio data is plentiful (from audiobooks, podcasts, etc.), high-quality labeled video data—where the speaker's mouth is clearly visible and synchronized with text—is scarce and expensive to collect. Furthermore, models often struggle with domain shift; a model trained on news broadcasts may fail on medical lectures due to differing vocabulary and linguistic patterns. Traditionally, adapting to a new domain requires collecting new labeled videos, which is often impractical.

Methods

OpenSR addresses these challenges through modality transfer. If audio and visual features are aligned into a shared phoneme space, a model learning to transcribe audio should inherently understand how to transcribe lips. The framework employs a three-stage pipeline:

  1. Pre-training (Multi-modal Alignment): A self-supervised model (e.g., AV-Hubert) maps unlabeled audio and video into a shared phoneme space.
  2. Audio-only Decoder Training: The pre-trained multi-modal encoder is frozen, and a Transformer decoder is trained using only labeled audio from the target domain. The frozen encoder ensures the shared phoneme space remains intact, allowing the decoder to learn the domain's syntax and vocabulary via audio and generalize this knowledge to visual inputs.
  3. Cluster-based Prompt Tuning: For "few-shot" scenarios with minimal video data, OpenSR introduces Cluster Prompt. Instead of full-model fine-tuning, which risks overfitting, small learnable embeddings are tuned for specific phoneme clusters to refine visual features without disrupting the learned language model.
Figure 1: The three-stage OpenSR training pipeline, illustrating pre-training, audio-only decoder learning, and the various inference modes (Zero-Shot, Few-Shot, Full-Shot).

Key Insights

A critical finding is the "Freezing" Requirement: the encoder must remain frozen during audio-only decoder training. Updating the encoder on audio-only data breaks the cross-modal alignment, destroying the model's ability to process visual data. By using audio as a proxy for video, the decoder gains a superior "language model" compared to training on limited video sets, as audio is easier to transcribe and contains richer linguistic information.

Results

OpenSR establishes new state-of-the-art (SOTA) benchmarks on the LRS2 and LRS3 datasets. On LRS2, it achieved a Word Error Rate (WER) of 2.7% for audio-visual speech recognition (AVSR) and 25.0% for lip-reading. Notably, the zero-shot lip-reading model (trained on 0 hours of labeled video) achieved a 36.0% WER on LRS2, outperforming several previous models trained on the full 224-hour labeled video set.

Figure 2: Comparison of OpenSR against previous methods. The zero-shot performance (36.0% WER) significantly outperforms existing models trained on substantial labeled video data.

Conclusion

OpenSR demonstrates that cross-modal alignment is a powerful tool for low-resource speech recognition. While the "homophene problem" (visually identical words like "pet" and "bet") remains a challenge, the framework's ability to leverage audio data for visual tasks represents a significant step forward. Future work will focus on tightening phoneme-level alignment and extending this "open-modality" approach to other paired signals like sign language.

Summary updated

Abstract

Speech Recognition builds a bridge between the multimedia streaming (audio-only, visual-only or audio-visual) and the corresponding text transcription. However, when training the specific model of new domain, it often gets stuck in the lack of new-domain utterances, especially the labeled visual utterances. To break through this restriction, we attempt to achieve zero-shot modality transfer by maintaining the multi-modality alignment in phoneme space learned with unlabeled multimedia utterances in the high resource domain during the pre-training \cite{shi2022learning}, and propose a training system Open-modality Speech Recognition (\textbf{OpenSR}) that enables the models trained on a single modality (e.g., audio-only) applicable to more modalities (e.g., visual-only and audio-visual). Furthermore, we employ a cluster-based prompt tuning strategy to handle the domain shift for the scenarios with only common words in the new domain utterances. We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and full-shot lip-reading methods. To the best of our knowledge, OpenSR achieves the state-of-the-art performance of word error rate in LRS2 on audio-visual speech recognition and lip-reading with 2.7\% and 25.0\%, respectively. The code and demo are available at https://github.com/Exgc/OpenSR.
BibTeX
@inproceedings{cheng2023opensr,
  author = {Xize Cheng and Tao Jin and Linjun Li and Wang Lin and Xinyu Duan and Zhou Zhao},
  title = {{OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.363},
}
Figure