u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality

Wei-Ning Hsu, Bowen Shi

u-HuBERT: Unified Mixed-Modal Speech Pretraining

While audio-visual speech models—which leverage lip-reading alongside sound—yield superior performance and robustness compared to audio-only models, their widespread adoption faces significant hurdles. u-HuBERT (Unified Hidden Unit BERT) is a self-supervised pre-training framework designed to tackle these challenges by learning from a mix of audio-only, video-only, and audio-visual speech data.

By utilizing a technique called "modality dropout" during training, u-HuBERT creates a single model capable of state-of-the-art performance across all modalities. Most impressively, it enables "zero-shot" transfer, allowing a model fine-tuned only on audio to successfully process visual-only speech.

The Challenge of Audio-Visual Speech

Building robust audio-visual models is traditionally difficult due to three main factors:

  1. Data Scarcity: Large-scale, high-quality paired audio-visual data is much harder to source than simple audio or text.
  2. Deployment Costs: Traditionally, deploying these systems requires maintaining different models for different scenarios (one for audio, one for video, one for both), which is computationally expensive.
  3. The Modality Gap: Models trained on one modality (e.g., audio) usually fail completely when presented with another (e.g., video), even though they are describing the exact same underlying speech.

The authors sought to create a versatile model capable of ingesting any combination of speech modalities, leveraging all available data to improve performance across the board.

The u-HuBERT Architecture

u-HuBERT builds on the success of HuBERT, a popular self-supervised speech model, but introduces critical innovations to handle mixed modalities.

u-HuBERT Architecture
  • Unified Architecture: The model uses modality-specific encoders (a ResNet-18 for video and a linear layer for audio filter banks) that feed into a shared Transformer backend.
  • Masked Cluster Prediction: Similar to BERT for text, u-HuBERT masks parts of the input and tries to predict "pseudo-labels." These labels are generated by clustering the model's own internal features from a previous iteration, creating a self-improving feedback loop.
  • Modality Dropout (The "Secret Sauce"): During pre-training, the model randomly "drops" either the audio or video stream. This forces the shared Transformer to learn modality-agnostic representations. Because the target labels remain the same regardless of the input source, the model learns that the sound of a word and the sight of someone saying it represent the same fundamental information.

Key Insights: Bridging the Modality Gap

The most significant finding of the paper is that modality dropout is essential for true multimodal understanding.

t-SNE Visualizations of Modality Alignment

Without dropping modalities during training, the model's internal representations for audio and video remain distinct and clustered separately. Modality dropout forces these representations into the same shared latent space.

This alignment unlocks Zero-Shot Generalization. Because the pre-training aligns audio and video so tightly, you can fine-tune the model using only labeled audio data, and it will still work reasonably well for lip-reading (visual-only) tasks. Furthermore, adding plentiful audio-only data during pre-training actually improves the performance of the model when it's later tested on audio-visual or visual-only inputs, elegantly solving the data scarcity problem.

Key Results

The researchers evaluated u-HuBERT on major datasets including LRS3, VoxCeleb2-En, TED-LIUM 3, and LibriSpeech. The results on the LRS3 benchmark are highly impressive:

  • State-of-the-Art Performance: A single u-HuBERT model achieved a Word Error Rate (WER) of 1.2% for audio-visual, 1.4% for audio-only, and 27.2% for visual-only input.
  • Zero-Shot Success: A model fine-tuned only on audio achieved a 31.6% WER on visual-only speech. While slightly higher than a specialized visual model, it is a massive improvement over previous methods that essentially failed (90%+ WER) in this zero-shot scenario.
  • Robustness: In noisy environments (e.g., 0dB babble noise), the audio-visual model maintained high accuracy where audio-only models typically degrade rapidly.

Limitations and Future Work

While groundbreaking, u-HuBERT has a few limitations. The current framework assumes there is an "anchor" modality (usually audio) that is paired with all other modalities at some point during training. It cannot yet bridge disconnected modalities that have never been paired together (e.g., Video and EMG data with no common audio link). Additionally, during fine-tuning, the model can sometimes suffer from "catastrophic forgetting" of its modality-agnostic nature if the fine-tuning data is too narrow, though freezing certain layers can mitigate this.

Future work involves expanding this framework to even more exotic speech modalities, such as Electromyography (EMG) or ultrasonic sensors, paving the way for a truly universal, modality-agnostic speech interface.

Summary updated

Abstract

While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality. In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective. By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models. Moreover, our model fine-tuned only on audio can perform well with audio-visual and visual speech input, achieving zero-shot modality generalization for multiple speech processing tasks. In particular, our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input.
BibTeX
@inproceedings{hsu2022u,
  author = {Wei-Ning Hsu and Bowen Shi},
  title = {{u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=zrAUoI2JA2},
}
Figure