GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions

Heda Zuo, Weitao You, Junxian Wu, Shihong Ren, Pei Chen, Mingxu Zhou, Yujia Lu, Lingyun Sun

Introduction

Composing the "perfect" background music for a video is one of the most subjective and complex challenges in artificial intelligence. A successful score must do more than just match a general "vibe"; it must synchronize with visual motion, reflect emotional shifts, and maintain rhythmic coherence. Traditional automated approaches have historically struggled with these nuances, often relying on rigid, rule-based systems (e.g., mapping fast motion to high BPM) or using Large Language Models (LLMs) as a descriptive bridge. While LLMs can capture semantic themes, they often strip away the granular temporal details necessary for precise synchronization.

In this study, we present GVMGen, a general video-to-music generation model designed to achieve robust music-video correspondence. By employing a hierarchical attention mechanism, GVMGen extracts and aligns visual features in both spatial and temporal dimensions, bypassing the limitations of text-based intermediaries to generate high-fidelity, multi-style audio.

The GVMGen Architecture

GVMGen utilizes an end-to-end encoder-decoder architecture specifically optimized for cross-modal alignment. The generation process is divided into three distinct stages:

  1. Visual Feature Extraction: The model employs a pretrained Vision Transformer (ViT) with spatial self-attention. By treating the video as a sequence of frames, the model preserves deep spatial details and the overarching temporal flow of the visual input.
  2. Feature Transformation: To bridge the gap between pixels and sound, GVMGen uses trainable music queries and spatial cross-attention. This creates an implicit "visual-musical" shared space, effectively filtering out redundant visual data and isolating the features most relevant to musical composition.
  3. Conditional Music Generation: The decoder, built upon the MusicGen architecture, utilizes temporal cross-attention to ensure the generated audio aligns with the specific timing of visual events. Unlike models restricted to MIDI format, GVMGen uses an Encodec-based decoder to produce high-fidelity waveform audio, capturing rich timbres and expressive nuances.

Overcoming Dataset Mismatch

A significant hurdle in video-to-music research is the nature of existing datasets. Many benchmarks rely on Music Videos (MVs), where the visual content is typically edited to match a pre-existing track. For background music generation, the requirement is reversed: the music must be composed to fit a fixed visual narrative.

To address this, the researchers compiled a new, large-scale dataset totaling 147 hours of video-music pairs across movies, vlogs, comics, and documentaries. Notably, the dataset includes 89.5 hours of Chinese Traditional Music (CTM), featuring instruments like the Erhu and Pipa. These instruments offer complex timbres and fingering techniques that are difficult to represent in MIDI, providing a rigorous test for the model’s generative richness and diversity.

Evaluation and Metrics

Standard metrics often fail to capture the nuances of music-video alignment. Consequently, the authors proposed a novel evaluation model along with two objective metrics:

  • Cross-Modal Relevance (CMR): Measures global style and thematic matching between the video and the generated audio.
  • Temporal Alignment (TA): Assesses local timing accuracy, ensuring that musical transitions correspond to visual cuts or motion shifts.

Experimental Results

Experimental evaluations demonstrate that GVMGen significantly outperforms existing baselines such as M2UGen and V2Meow.

  • Audio Quality: GVMGen achieved a Fréchet Audio Distance (FAD) of 7.45, the lowest among tested models, indicating superior audio fidelity.
  • Correspondence: The model reached a CMR of 0.88, proving a high degree of relevance to the input video.
  • Subjective Testing: In listening tests, both expert and non-expert listeners consistently rated GVMGen higher in terms of Overall Music Quality (OMQ) and Music-Video Correspondence (MVC).
  • Zero-Shot Universality: Remarkably, GVMGen demonstrated strong performance on entirely unseen datasets (such as SymMV) without additional fine-tuning, highlighting its potential as a general-purpose tool for content creators.

Conclusion and Future Work

GVMGen represents a significant step forward in automated scoring by prioritizing implicit spatial-temporal alignment over explicit language bridges. While the model excels at general correspondence, future work will focus on enhancing user-driven personalization—allowing creators to specify sub-styles or specific instruments via prompts—and refining "mickey-mousing" capabilities for extremely rapid visual-audio synchronization. By providing both a robust model and a diverse new dataset, GVMGen sets a new benchmark for the future of generative media.

Summary updated

Abstract

Composing music for video is essential yet challenging, leading to a growing interest in automating music generation for video applications. Existing approaches often struggle to achieve robust music-video correspondence and generative diversity, primarily due to inadequate feature alignment methods and insufficient datasets. In this study, we present General Video-to-Music Generation model (GVMGen), designed for generating high-related music to the video input. Our model employs hierarchical attentions to extract and align video features with music in both spatial and temporal dimensions, ensuring the preservation of pertinent features while minimizing redundancy. Remarkably, our method is versatile, capable of generating multi-style music from different video inputs, even in zero-shot scenarios. We also propose an evaluation model along with two novel objective metrics for assessing video-music alignment. Additionally, we have compiled a large-scale dataset comprising diverse types of video-music pairs. Experimental results demonstrate that GVMGen surpasses previous models in terms of music-video correspondence, generative diversity, and application universality.
BibTeX
@inproceedings{zuo2025gvmgen,
  author = {Heda Zuo and Weitao You and Junxian Wu and Shihong Ren and Pei Chen and Mingxu Zhou and Yujia Lu and Lingyun Sun},
  title = {{GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=2kma3WUSbE},
}
Figure