From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation

Kun Su, Xiulong Liu, Eli Shlizerman

Abstract

In current artificial intelligence research, the abilities to understand audio-visual content (representation learning) and to create it (generative modeling) have largely evolved in isolation. This paper introduces VAB (Vision to Audio and Beyond), a unified framework designed to bridge this gap. By performing masked token prediction within a shared latent space, VAB achieves high-quality video-to-audio generation with a 17x speedup over existing autoregressive models. Simultaneously, the framework sets new benchmarks in cross-modal retrieval and classification, demonstrating that generative pre-training significantly enhances a model's representational understanding of the link between sight and sound.

Introduction

When observing a silent video of a glass shattering or a dog barking, human cognition naturally "hallucinates" the missing auditory component. Replicating this cross-modal synergy in AI has proven difficult. Generative models are typically computationally expensive and specialized, while representation models—used for tasks like searching videos via audio—are often purely discriminative, lacking the ability to synthesize new content. Furthermore, the high dimensionality of raw audio and video data makes joint modeling a significant engineering challenge.

VAB addresses these issues by proposing a single, efficient architecture that handles both representation and generation. By moving away from raw pixels and waveforms into a compressed latent space, VAB provides a scalable solution for multi-modal reasoning.

The VAB Framework

Latent Space Tokenization

The core innovation of VAB is its reliance on discrete tokens rather than raw signals. This drastically reduces the sequence length the model must process:

  • Visual Encoding: VAB utilizes a pre-trained eva-CLIP image encoder to extract visual features at a rate of 1 frame per second (fps). A 10-second video is thus reduced to just 10 visual tokens.
  • Audio Encoding: A neural audio codec (such as DAC or Encodec) transforms raw audio into discrete tokens, allowing the model to treat audio generation as a sequence prediction task.

Multiway Transformer Encoder

The architecture consists of a 24-layer transformer designed to balance modality-specific nuances with joint reasoning. The first 12 layers act as "modal-specific experts," dedicated to processing either vision or audio independently. The subsequent 12 layers are shared, allowing the model to fuse these features for cross-modal tasks. This structure prevents the longer audio sequences from "drowning out" the visual features during the learning process.

Training and Parallel Decoding

VAB is trained on visual-conditioned masked audio token prediction. During training, the model receives video features and a partially masked set of audio tokens; its task is to predict the missing auditory information.

For inference, VAB employs an iterative parallel decoding strategy. Unlike traditional autoregressive models that generate audio one token at a time, VAB starts with a full sequence of [MASK] tokens and fills them in simultaneously over a few iterations. This approach is the primary driver behind the model's significant inference speed.

Experimental Results

Generation Performance and Speed

VAB demonstrates a remarkable balance between quality and efficiency. In benchmarks using the VGGSound dataset, VAB achieved a Fréchet Audio Distance (FAD) of 2.69, outperforming several specialized diffusion and autoregressive baselines. More notably, the VAB-Encodec variant can generate 10 seconds of high-quality audio in just 0.4 seconds. Compared to FoleyGen, which requires 6.9 seconds for the same task, VAB offers a 17x speedup.

Representation and Retrieval

The model's unified nature allows it to excel in discriminative tasks. VAB significantly outperformed general-purpose models like ImageBind and LanguageBind in video-to-audio retrieval. On the AudioSet and VGGSound benchmarks, VAB often doubled the "Recall@1" accuracy of its competitors, proving that generative training serves as a powerful "teacher" for semantic understanding.

Human Evaluation

In side-by-side subjective tests, human raters consistently preferred VAB-generated audio over baseline models. The results indicated higher scores in both overall acoustic quality and relevance to the visual content.

Discussion and Limitations

While VAB represents a significant step toward unified audio-visual AI, several challenges remain:

  • Temporal Resolution: The 1fps visual sampling rate, while efficient, makes fine-grained synchronization (such as lip-syncing) difficult.
  • Spatial Awareness: Because the model uses global CLIP features, it lacks the ability to localize sound sources within a frame. For example, the audio does not currently pan to follow a car moving from left to right.
  • Data Noise: Training on large-scale YouTube datasets (AudioSet and VGGSound) means the model occasionally inherits "in-the-wild" background noise that may be irrelevant to the primary visual subject.

Conclusion

VAB demonstrates that a single transformer architecture can effectively master both the representation and generation of audio-visual data. By leveraging latent space tokenization and parallel decoding, the framework provides a high-speed, high-quality solution for cross-modal synthesis. Future iterations of the model will likely focus on increasing visual frame rates and incorporating spatial features to achieve more precise, localized audio-visual alignment.

Summary updated

BibTeX
@inproceedings{su2024from,
  author = {Kun Su and Xiulong Liu and Eli Shlizerman},
  title = {{From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=jU6iPouOZ6},
}
Figure