Learning Cross-Lingual Visual Speech Representations

Andreas Zinonos, Alexandros Haliassos, Pingchuan Ma, Stavros Petridis, Maja Pantic

Introduction

Visual Speech Recognition (VSR), commonly known as lip-reading, is the process of interpreting speech solely from visual cues. This task is inherently difficult because many distinct sounds (phonemes) are visually indistinguishable (visemes). While deep learning has significantly advanced VSR, high-performing models typically require thousands of hours of labeled data. Consequently, research has been heavily biased toward English, leaving many other languages with subpar performance due to data scarcity.

This paper investigates the potential of self-supervised learning (SSL) to bridge this gap. By leveraging large-scale unlabeled video data, the researchers explore whether it is possible to learn "universal" visual speech representations that transfer across different languages, potentially reducing the reliance on massive labeled datasets for non-English speech.

The RAVEn Framework

The study utilizes the RAVEn (Raw Audio-Visual Speech Encoders) framework, a self-supervised approach based on a student-teacher architecture.

Architecture and Pre-training

The model comprises a video encoder (ResNet-18 combined with a Transformer) and an audio encoder (1D ResNet-18 and a Transformer). During the pre-training phase, the "student" model receives masked inputs—where portions of the audio or video are hidden—and is tasked with predicting the representations generated by the "teacher" model, which processes the full, unmasked input. The optimization process employs both cross-modal (audio-to-video and vice-versa) and within-modal (audio-to-audio) cosine similarity losses.

Fine-tuning

Once pre-training is complete, the audio encoder is discarded. The pre-trained visual backbone is then attached to a Transformer decoder. This combined model is fine-tuned on labeled data using a joint Connectionist Temporal Classification (CTC) and attention loss to perform the final lip-reading task.

Experimental Setup

Datasets

  • Pre-training: The authors used AVSpeech, a multilingual dataset containing approximately 4,700 hours of video. A language classifier was employed to segment the data into English, Portuguese, Spanish, French, Italian, and Mandarin.
  • Fine-tuning & Evaluation: The models were evaluated on English (LRS3), Spanish, Italian, Portuguese, and French (Multilingual TEDx), and Mandarin (CMLR).

Key Insights and Analysis

The Multilingual Advantage

The research demonstrates that for low-resource languages, pre-training on a diverse multilingual dataset is significantly more effective than pre-training on the target language alone. This suggests that the model learns fundamental structures of human speech and facial movement that are shared across linguistic boundaries.

The "Dilution" Effect

A notable finding was the performance trade-off in high-resource scenarios. For English, which already possesses a vast amount of training data, the addition of other languages to the pre-training set resulted in a slight increase in the error rate. This suggests that when data is abundant, language-specific specialization may be more beneficial than broad, cross-lingual features.

Linguistic Proximity and Generalization

The study found that the similarity between languages impacts the effectiveness of transfer. For instance, adding Portuguese data to a Spanish model provided a greater performance boost than adding Mandarin. Furthermore, the models showed impressive robustness to unseen languages, performing competitively even when fine-tuned on a language not included in the pre-training phase.

Results

The primary metric for evaluation was the Character Error Rate (CER). The results highlight the transformative impact of multilingual pre-training:

  • Significant Gains in Low-Resource Languages: In Italian, a monolingual model achieved a CER of 76.9%. However, the multilingual model reduced this error rate to 38.0%.
  • Quality vs. Quantity: When the total pre-training time was held constant at 1,333 hours, a multilingual model (with reduced English data) outperformed an English-only model on all non-English tasks. In Spanish, the CER dropped from 35.9% (English-only) to 32.8% (Multilingual).
  • Data Density Requirements: In extremely restricted settings (e.g., only 30 hours of pre-training), monolingual models still performed better, indicating that a certain "density" of data per language is necessary for the benefits of multilingual learning to manifest.

Conclusion

This research proves that self-supervised, cross-lingual pre-training is a powerful tool for improving VSR in languages beyond English. While there is a slight performance trade-off for high-resource languages like English, the gains for low-resource languages are substantial. Future work may focus on better balancing techniques to mitigate the dilution effect and exploring how specific visemes map across different language families to create truly universal visual speech representations.

Summary updated

Abstract

Cross-lingual self-supervised learning has been a growing research topic in the last few years. However, current works only explored the use of audio signals to create representations. In this work, we study cross-lingual self-supervised visual representation learning. We use the recently-proposed Raw Audio-Visual Speech Encoders (RAVEn) framework to pre-train an audio-visual model with unlabelled multilingual data, and then fine-tune the visual model on labelled transcriptions. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance; (2) multi-lingual outperforms English-only pre-training; (3) using languages which are more similar yields better results; and (4) fine-tuning on unseen languages is competitive to using the target language in the pre-training set. We hope our study inspires future research on non-English-only speech representation learning.
BibTeX
@inproceedings{zinonos2023learning,
  author = {Andreas Zinonos and Alexandros Haliassos and Pingchuan Ma and Stavros Petridis and Maja Pantic},
  title = {{Learning Cross-Lingual Visual Speech Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=5uWlYIYZFh},
}
Figure