Training a model to simultaneously understand or generate audio and video is a fundamentally asymmetric challenge. On one hand, single-modality datasets are massive: the internet provides millions of hours of uncurated speech, text, and video. On the other hand, perfectly synchronized, high-fidelity, paired audio-visual data is exceptionally scarce.
High-quality audio requires studio conditions, while high-quality video requires precise capture and alignment. When sourced from the wild, audio is often noisy, video is low-resolution, or the two streams are completely unaligned. This cross-modal data scarcity creates a severe bottleneck across a wide range of tasks, from speech recognition and visual dubbing to spatial localization and sentiment analysis.
To bypass this bottleneck, researchers cannot simply rely on scaling up raw data collection. Instead, they are developing architectural mechanisms designed to learn from unpaired, heterogeneous, or synthetic data sources.
The Scale Disparity of Multimodal Data
The core problem stems from the logistical hurdles of collecting synchronized ground truth. As noted by Andrew Rouditchenko et al. in their work on Whisper-Flamingo, while audio-only models benefit from hundreds of thousands of training hours, high-quality lip-reading data remains a small fraction of that scale.
This scarcity manifests in several distinct ways across the literature:
- Longitudinal and Contextual Scarcity: In video analysis, human-annotated labels for combined audio-visual sentiment across long-form media are rare, forcing researchers like Eric Chu et al. to rely on single-modality data for mapping emotional trajectories in Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies.
- Cross-Domain Mapping Gaps: Tasks that map physical appearance to vocal traits—such as AnyoneNet by Qicong Xie et al.—suffer from a shortage of datasets pairing static facial images with high-fidelity audio. Similarly, Nikhil Singh et al. point out in Image2Reverb that the rarity of paired images and acoustic impulse responses forces a reliance on weak supervision.
- Spatial and Environmental Limitations: For autonomous navigation, Praneeth Chakravarthula et al. note in Seeing With Sound that existing datasets lack the synchronized multi-channel audio and 3D spatial annotations needed for dynamic urban environments.
When naturally synchronized streams are missing, researchers are often forced to synthetically pair unrelated data. However, as argued by Shuncheng Jia et al. in Explaining cocktail party effect and McGurk effect with a spiking neural network improved by Motif-topology, these artificial pairings can fail to capture the nuanced complexities of real-world multisensory integration.
Distilling Knowledge from Single-Modality Giants
If paired data is scarce, how can a model learn the complex correlations between sight and sound? One powerful solution is cross-modal knowledge distillation. Instead of training an audio-visual model from scratch, researchers leverage massive, pre-trained single-modality foundation models that have already mapped out the structure of a single domain.
[Massive Audio-Only Data] ---> [Pre-trained Speech Foundation Model (Teacher)]
|
v (Knowledge Distillation)
[Scarce Paired Data] ---> [Audio-Visual Student Model]
^
| (Cross-Modal Alignment)
[Massive Video-Only Data] ---> [Pre-trained Visual Encoder]
In Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models, Jing-Xuan Zhang et al. demonstrate this by transferring the rich representational space of large-scale Speech Foundation Models (SFMs) into a student model capable of both seeing and hearing. Because the teacher model has already learned robust speech representations from millions of hours of raw audio, the student only needs to learn how to map visual features (like lip movements) into that pre-established latent space, drastically reducing the amount of paired data required.
A similar intuition drives Whisper-Flamingo. Instead of co-training audio and video architectures, Andrew Rouditchenko et al. freeze OpenAI's pre-trained Whisper model and integrate visual lip-reading features through a gated cross-attention mechanism. By freezing the weights of the underlying foundation models, the network preserves its robust speech recognition capabilities while learning cross-modal alignment from a relatively small multimodal dataset.
Decoupling and Disentangling Pipelines
End-to-end models typically require perfectly synchronized, high-resolution pairs because they attempt to map raw pixels directly to raw waveforms. A prominent strategy for overcoming data scarcity is architectural decoupling: breaking the pipeline into independent stages so that the model can learn different facets of the problem from imperfect or heterogeneous data.
For example, the authors of Towards Realistic Visual Dubbing with Heterogeneous Sources split the visual dubbing task into two distinct stages: predicting lip movements and generating the final image. This separation allows the model to learn the structural dynamics of speech from low-resolution or silent web videos, while reserving high-resolution, identity-preserving generation for static images.
[Low-Res / Silent Video] ---> [Stage 1: Lip-Movement Prediction] ---\
v
[Static / High-Res Image] -> [Stage 2: Identity-Preserving Rendering] -> [High-Fidelity Dubbed Video]
In generative tasks, this decoupling prevents data scarcity in one modality from degrading the quality of another. Songju Lei et al. design Uni-Dubbing to synthesize speech from silent lip movements using a zero-shot approach. By freezing pre-trained multimodal encoders and utilizing discrete neural units, they decouple the speech generation from the target language, eliminating the need for paired audio-visual data in the target language entirely.
Similarly, Shengeng Tang et al. bypass the need for pristine audio-visual training pairs in Text2Lip by driving facial animation directly from text via viseme-guided rendering. This design choice makes the model immune to the acoustic noise and homophene ambiguities that plague audio-driven animation models when clean training data is unavailable.
For video-to-audio synthesis (Foley generation), Ho Kei Cheng et al. introduce MMAudio, which trains a multimodal transformer jointly on both scarce audio-visual pairs and abundant audio-text data. This allows the model to learn complex semantic concepts from text descriptions while reserving the limited audio-visual data purely for learning precise temporal synchronization.
Pseudo-Pairing and Synthetic Bridges
When real-world paired data is completely unavailable, researchers often use a third, highly abundant modality—such as language—as a semantic bridge to create pseudo-paired datasets.
This mechanism is central to the work by Julia Wilkins et al. in Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries. Professional sound production requires high-fidelity audio, but most public audio-visual datasets (like YouTube clips) feature low-quality, uncurated sound. To bridge high-quality isolated sound effects with stock video, the authors use Large Language Models (LLMs) and CLIP to generate text descriptions for both modalities. By aligning these descriptions in a shared text-embedding space, they create a high-fidelity pseudo-paired dataset without requiring actual synchronized recordings.
[Stock Video] ---> [CLIP Text Description] ---\
v
[Shared Semantic Space]
^
[HQ Sound Effect] ---> [LLM Text Description] ---/
In a similar vein, Xin Cheng et al. develop VSSFlow to unify environmental sound and visual text-to-speech generation. Recognizing that joint datasets containing perfectly synchronized speech and environmental sounds are extremely rare, they use a flow-matching Diffusion Transformer with a disentangled conditioning mechanism, relying on synthetic data generation to fill the cross-modal gaps.
Missingness-Resilient and Adaptive Fusion
Data scarcity does not just affect training; it also impacts inference, where one or more modalities may be corrupted or entirely missing. If a model over-relies on a rigid cross-modal correlation learned during training, it will fail when deployed in real-world, noisy environments.
To address this, Payal Mohapatra et al. introduce a weight-sharing architecture in Missingness-resilient Video-enhanced Multimodal Disfluency Detection. By forcing the audio and visual processing streams to share weights, the network learns a generalized representation space. If the video stream drops out entirely, the audio stream can still leverage the shared parameters, outperforming traditional audio-only baselines despite the missing modality.
Alternatively, models can use adaptive weighting mechanisms to dynamically shift reliance between modalities based on signal quality. In Multi-target DoA Estimation with an Audio-visual Fusion Mechanism, Maulik C. Madhavi et al. develop a system for multi-speaker localization. Because high-quality datasets with synchronized multi-channel audio and 3D spatial video are rare, models must be flexible. Their adaptive fusion mechanism intelligently downweights the visual stream when a speaker moves off-camera or when the video is uninformative, ensuring that the system degrades gracefully rather than failing catastrophically.
Curating the Next Generation of Ground Truth
While algorithmic workarounds like distillation and decoupling are crucial, the long-term advancement of the field depends on expanding the scale of available ground truth. Consequently, a parallel track of research focuses on curating massive, tightly synchronized benchmarks designed to push past current data limitations.
Several recent benchmarks aim to fill these specific modal gaps:
- Text-Audio-Video Alignment: Xuyang Shen et al. introduced TAVGBench, a dataset of 1.7 million synchronized clips paired with high-quality text descriptions, designed to train models on the intricate mappings between visual events and acoustic signatures.
- Scene-Aware Atmosphere: To study how environment affects speech, Jiawei Jin et al. released Vivid-210K alongside VividVoice, providing the synchronized triplets of speaker identity, clean speech, and environmental context necessary for scene-aware audio synthesis.
- Trimodal Summarization: Sumin Kim et al. developed the MoSu benchmark for TripleSumm, providing continuous, synchronized visual, audio, and textual features for video summarization.
- Long-Form Conversational Audio-Visual Data: Addressing the "boss level" challenge of overlapping speech in meetings, Chao-Han Huck Yang et al. and Ming Gao et al. introduced the MISP-Meeting corpus for the MISP 2025 Challenge. This dataset combines 360-degree video with multi-channel audio, proving that when synchronized visual cues like lip movements are explicitly provided, models can resolve severe acoustic ambiguities and drastically reduce transcription error rates.
Key Foundational Work
Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild (2017) — contributed 1 concept elements to this direction.
Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies (2017) — contributed 1 concept elements to this direction.
Paying Attention to Descriptions Generated by Image Captioning Models (2017) — contributed 1 concept elements to this direction.