From Sound to Sight: Towards AI-authored Music Videos

Leo Vitasovic, Stella Graßhof, Agnes Mercedes Kloft, Ville V. Lehtola, Martin Cunneen, Justyna Starostka, Glenn McGarry, Kun Li, Sami S. BrandtarXiv 2025 · DOI · arXiv

Introduction

Traditional music visualization has historically been limited to reactive, signal-driven patterns—think of the oscillating waveforms and abstract shapes of early media players. While these visualizations are rhythmically synchronized, they lack semantic depth. They fail to capture the "mood" of a melancholy piano piece or the narrative arc of an orchestral composition. Recent research aims to bridge this gap by moving beyond simple signal processing toward automated, cinematic storytelling.

The AI "Creative Director"

To solve the challenge of computational synaesthesia, researchers have developed a system that uses Large Language Models (LLMs) as intermediary "creative directors." Instead of mapping audio directly to pixels, the system first "listens" to the music to understand its emotional and instrumental nuances, scripts a visually coherent story, and then generates video clips based on that script.

The automated pipeline for transforming song inputs into concatenated video clips.

Proposed Pipelines

The study explores two distinct methodologies for this translation:

  1. CLAP-based Pipeline: This approach uses Contrastive Language-Audio Pre-training (CLAP) to extract semantic labels (e.g., "upbeat tempo," "melodic strings"). These labels are processed by an LLM (DeepSeek-R1-Distill-Llama-8B) to write a scene-by-scene narrative, which is then rendered by the Mochi-1 video model.
  2. LALM-based Pipeline: This employs a Large Audio Language Model (LALM) to interpret raw audio directly into a story arc. The LLM then generates segment-aligned visual prompts for the Wan 2.1 video model.

Both pipelines utilize a rule-based segmentation system that mimics a human editor, making cuts based on frequency shifts, beat counts, or duration limits to ensure rhythmic alignment.

Key Insights and Results

A significant finding of the research is that the CLAP-based approach—which relies on explicit semantic labeling—outperformed the more integrated LALM approach in user trials. On a 7-point scale, the CLAP pipeline achieved a higher overall rating (M=2.93) compared to the LALM pipeline (M=2.64). This suggests that explicitly "naming" musical qualities helps the LLM generate more effective and evocative visual prompts.

Examples of generated visual narratives across diverse musical genres, from Traditional Folk to Ambiental Techno.

Users noted that the AI was particularly adept at capturing the general mood and emotional alignment of the music, even without access to lyrics. However, the system still faces the "stock footage trap," where generated clips can feel like a collection of unrelated shots rather than a unified film.

Challenges and Future Directions

While the pipelines represent a leap forward in automated creativity, several technical hurdles remain:

  • Visual Inconsistency: Maintaining character and setting consistency across multiple shots is difficult; future iterations may use "character loRA" techniques to lock in specific visual identities.
  • Stylistic Control: The videos often lack a unifying artistic vision. Researchers suggest that using persistent stylistic prompts or predefined color palettes could improve aesthetic coherence.
  • Lyric Integration: Although the current goal was to work without lyrics, incorporating text data would likely deepen the narrative understanding of the track.

Ultimately, this research raises fundamental questions about the degree of AI agency in art—asking whether these systems are merely sophisticated instruments or the forerunners of autonomous digital creators.

Summary updated

Abstract

Conventional music visualisation systems rely on handcrafted ad hoc transformations of shapes and colours that offer only limited expressiveness. We propose two novel pipelines for automatically generating music videos from any user-specified, vocal or instrumental song using off-the-shelf deep learning models. Inspired by the manual workflows of music video producers, we experiment on how well latent feature-based techniques can analyse audio to detect musical qualities, such as emotional cues and instrumental patterns, and distil them into textual scene descriptions using a language model. Next, we employ a generative model to produce the corresponding video clips. To assess the generated videos, we identify several critical aspects and design and conduct a preliminary user evaluation that demonstrates storytelling potential, visual coherency and emotional alignment with the music. Our findings underscore the potential of latent feature techniques and deep generative models to expand music visualisation beyond traditional approaches.
BibTeX
@misc{vitasovic2025from,
  author = {Leo Vitasovic and Stella Graßhof and Agnes Mercedes Kloft and Ville V. Lehtola and Martin Cunneen and Justyna Starostka and Glenn McGarry and Kun Li and Sami S. Brandt},
  title = {{From Sound to Sight: Towards AI-authored Music Videos}},
  year = {2025},
  eprint = {2509.00029},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.00029},
}
Figure