Audio-visual speech translation means turning video of a person speaking into text in another language, using both the sound and the lip movements. It needs rare "triplets": a video clip, its transcript, and a translation. Today's best systems start from a model pretrained by masked prediction, where parts of the audio and lip video are hidden and the model learns to fill them in. They then fine-tune this model on the few triplets that exist. The pretrained model squeezes everything into one representation: what is being said, but also who is saying it, how the face is lit, and what noise is in the room. The translation decoder must then learn to ignore those nuisance factors from very little data.
We propose to change the masked-reconstruction task so that it splits the representation in two. One stream holds the speech content, which both modalities share. The other holds a single per-clip vector for speaker and scene, the private part. Hidden lip patches are reconstructed from the content stream computed from the audio, combined with the private vector taken from a different clip of the same speaker. This holds in the other direction too: hidden audio spans are reconstructed from the video-derived content stream. Because the private vector comes from another utterance, it cannot carry this utterance's words. Because the content stream must explain the other modality, it cannot carry this face's appearance. The translation decoder reads only the content stream. Our central prediction is that this split matters most where triplets are scarce and audio is noisy, and that the advantage over a standard pretrained model should grow as the number of translation triplets shrinks.
Core Claim
Shared-private decomposition separates what several inputs have in common (the shared part) from what is specific to each (the private part). It has usually been enforced with penalties. An adversarial discriminator can strip domain cues from the shared features, as in Domain Separation Networks and Multinomial Adversarial Networks. An orthogonality penalty on the product of shared and private features is another option, as in ASP-MTL. The file flags both as unstable and sensitive to tuning. DMVAE instead enforces the split by cross-reconstruction: it decodes modality B from modality A's shared code plus B's private code. On small benchmarks it reported 88.1% versus 9.5% for an undecomposed multimodal VAE when generating MNIST digits from SVHN images.
Our claim is that cross-reconstruction becomes a strong and stable way to split the representation when it is embedded inside high-ratio masked modeling, under two conditions. The private code must be pooled over time from a different clip of the same speaker. The shared code must come from the other modality. Under these conditions the split is enforced by what information can reach the decoder, not by a penalty weight.
DMVAE supplies the cross-reconstruction idea, but it uses shallow VAEs on toy data and has no masking or translation. Masked image modeling supplies the pretext task. Patch reconstruction keeps fine structure that supervised targets discard (BEiT, A Unified View of Masked Image Modeling). Triplet scarcity is the regime where we expect the payoff.
Relation to Prior Work
The closest systems are masked audio-visual pretraining models.
- AV-HuBERT masks audio and lip-video spans and predicts audio-derived cluster IDs. It reached 32.5% word error rate on LRS3 lip reading with 30 labeled hours, using modality dropout (randomly removing a whole input stream) to avoid relying on audio alone.
- VATLM extends this to text with a unified unit vocabulary.
- MuAViC fine-tunes AV-HuBERT for multilingual audio-visual speech translation and is our main benchmark.
All three use one monolithic backbone, so speaker appearance, lighting and noise share the representation with content. We keep their masking and their data and change only what the reconstruction is conditioned on.
Two speech translation methods motivate the need for structure under scarcity but act only at the output or loss level.
- TDA makes the transcribe-then-translate and translate-then-back-transcribe probability paths agree, with +1.8 BLEU on MuST-C.
- FuseST stacks contrastive, attention and distillation losses so that the model keeps using speech rather than leaning on the transcript.
Neither touches internal factorization. Both also use audio and text only.
MAM shows that masked spectrogram reconstruction alone helps speech translation by +1.1 BLEU, at 6.5% parameter overhead. It is the audio-only ancestor of our audio branch.
ZeroMMT attacks triplet scarcity for image-grounded translation with visually conditioned masked language modeling. It leaves the visual side frozen and unmasked. We do the opposite: we mask and reconstruct the visual input, in the speech setting.
Research Program
The model has three parts:
- Encoders. A lip-region video encoder and a spectrogram encoder. Each produces a per-frame shared stream $s$ and a single per-clip private vector $p$, obtained by attention pooling over unmasked frames.
- Light decoders. One reconstructs masked lip patches as pixels, the other masked spectrogram spans. Each takes a shared stream and a private vector.
- Translation decoder. It attends only to the fused shared streams.
Pretraining uses LRS3 and VoxCeleb2, as AV-HuBERT does. VoxCeleb2's speaker labels let us sample, for each clip $A$, a different clip $B$ from the same speaker. The pretraining objective is
$$ \mathcal{L} = \lVert M_v \odot (V_A - D_v(s^{a}_A, p^{v}_B)) \rVert^2 + \lVert M_a \odot (X_A - D_a(s^{v}_A, p^{a}_B)) \rVert^2 + \lambda\, \mathcal{L}_{\text{unit}}. $$
The terms are:
- $V_A$ and $X_A$ are the lip video and spectrogram of clip $A$.
- $M_v$ and $M_a$ are masks selecting hidden patches (about 75%) and hidden spans.
- $s^{a}_A$ is the shared stream computed from $A$'s audio. $s^{v}_A$ is the shared stream from $A$'s video.
- $p^{v}_B$ and $p^{a}_B$ are private vectors from clip $B$ of the same speaker.
- $D_v$ and $D_a$ are the reconstruction decoders.
- $\mathcal{L}_{\text{unit}}$ is AV-HuBERT's cluster-prediction loss on the fused shared stream, weighted by $\lambda$. It keeps the shared stream phonetic.
For the audio side, MUSAN noise is added to $A$. The audio decoder reconstructs $A$'s noisy spectrogram from video content plus the private vector of a clip $B$ carrying the same noise draw. This should push noise into the private side rather than the content stream.
Fine-tuning attaches a translation decoder to the shared streams. It is trained on MuAViC triplets subsampled to 5%, 25% and 100% of the available hours per language pair.
We will also measure whether the split happened, using linear probes:
- speaker-identity accuracy on the shared stream, which should approach chance;
- speaker-identity accuracy on the private vector, which should be high;
- phone or word decoding from the private vector, which should be near chance;
- the same probes on noise-type classification.
Baselines
The naive reference point is the end-to-end model trained from scratch on the subsampled MuAViC triplets with no pretraining. It shows how hard each scarcity level is.
The strongest existing method is the MuAViC recipe: AV-HuBERT Large pretrained with modality dropout, then fine-tuned for translation with noise augmentation, as in MuAViC. It is the strongest competitor because it is the only published system on this benchmark. It already includes the obvious robustness tricks.
MuAViC reports, under babble noise at a signal-to-noise ratio of 0, average BLEU of 7.8 for bilingual and 6.6 for multilingual audio-visual models. Whisper reaches 0.3 in the same condition, and the audio-visual models average +3.2 and +3.3 BLEU over audio-only models. For clean-condition audio-visual translation BLEU per language, no reported result found on Lacuna for MuAViC clean AVST in this file. We will therefore reproduce clean numbers ourselves.
For a fair comparison, we will reimplement this baseline at the same model size and on the same pretraining hours. We will match pretraining steps and accelerator-hours. The fine-tuning tuning budget will be identical, with the same learning-rate sweep and number of trials per method.
VATLM is a secondary baseline for the representation itself. It reported 8.4% relative lip-reading error reduction over AV-HuBERT at 30 hours. We run it on lip reading only, since it adds text data we do not use.
Two ablations isolate the key ingredient:
- Penalty instead of structure. Split shared and private streams in AV-HuBERT, but enforce the split with an orthogonality penalty plus a speaker-adversarial head in the ASP-MTL style, with no speaker swap. This tests whether the swap, rather than the split, matters.
- Same-clip private code. Take $p$ from the same clip $A$ instead of $B$. This tests whether cross-clip sourcing is what keeps content out of the private code.
A third ablation, replacing pixel targets with cluster targets in the video branch, tests whether masked image reconstruction is needed at all.
The proposal beats the baseline if both of the following hold:
- It improves noisy BLEU by a margin larger than seed variance, which we will measure over three seeds.
- The gain at 5% triplets is clearly larger than at 100%.
Matching the baseline at 100% while winning at 5% would still support the claim. Winning nowhere would refute it.
Why not just…?
Why not just rely on modality dropout and noise augmentation, as AV-HuBERT and MuAViC already do? Modality dropout makes each stream usable alone. Nothing in it removes speaker or noise information from the stream that is kept. MuAViC's own numbers show that even with audio-visual input, noisy speech recognition stays at 43.1% word error rate. This is a real threat, because AV-HuBERT's cluster targets already discard some speaker information. The head-to-head comparison at matched compute settles it, together with the speaker probe on each model's representation. If AV-HuBERT's features are already speaker-blind and the BLEU curves coincide, our premise is wrong.
Why not just add an orthogonality or adversarial penalty to a shared-private split? That is the textbook approach. The file notes it needs delicate tuning and that orthogonality does not stop private modules from also encoding shared content. Ablation 1 is exactly this alternative, given the same tuning budget. If it matches the speaker-swap model, we will report that the cheaper penalty suffices and that our contribution reduces to pairing it with masked pretraining.
Why not just make more triplets with machine translation, as MuAViC does with M2M-100? Pseudo-translations add synthetic errors, and synthetic data alone has historically not closed the gap to cascades (Sperber et al., Pino et al.). Synthesis is also complementary rather than competing. We will run both models with and without added M2M-100 pseudo-triplets. If pseudo-triplets erase our low-resource advantage, scarcity is not the right framing for this method.
Why not just use a cascade of speech recognition followed by text translation? In clean conditions a strong cascade may well win, and we will report it. Under babble noise, MuAViC shows Whisper falling to 0.3 BLEU. The noisy, low-triplet regime is where we claim an advantage, and we will not claim one elsewhere.
Proposed Experiments
The main experiment produces BLEU-versus-triplet-fraction curves on MuAViC:
- three triplet fractions: 5%, 25% and 100%;
- clean audio and MUSAN babble at signal-to-noise ratios of 0 and 5;
- bilingual and multilingual settings;
- three seeds per configuration.
It compares the proposal, the AV-HuBERT/MuAViC baseline, the two ablations and the from-scratch reference.
The second experiment is LRS3 lip reading with 30 labeled hours. It checks that the shared stream is not weaker for recognition, against the 32.5% word error rate reported by AV-HuBERT.
The third experiment is the probe suite: speaker and noise identity versus content, read from each stream.
The decisive check combines these results. The idea succeeds only if three things hold at once:
- the shared stream is near-chance for speaker identity;
- the private vector is near-chance for content;
- the noisy BLEU gain over AV-HuBERT widens as triplets shrink.
If the probes show a clean split but BLEU does not move, decomposition is not what limits low-resource audio-visual translation. That negative result would itself be worth reporting.
References
- Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
- Multinomial Adversarial Networks for Multi-Domain Text Classification
- Adversarial Multi-task Learning for Text Classification
- Private-Shared Disentangled Multimodal VAE for Learning of Latent Representations
- BEiT: BERT Pre-Training of Image Transformers
- A Unified View of Masked Image Modeling
- Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
- VatLM: Visual-Audio-Text Pre-Training With Unified Masked Prediction for Speech Representation Learning
- MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
- Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement
- Improving Speech Translation by Fusing Speech and Text
- MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation
- Towards Zero-Shot Multimodal Machine Translation
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are
- Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade