Automatic Realistic Music Video Generation from Segments of Youtube Videos

Sarah Gross, Xingxing Wei, Jun Zhu

Abstract

A Music Video (MV) is a video aiming at visually illustrating or extending the meaning of its background music. This paper proposes a novel method to automatically generate, from an input music track, a music video made of segments of Youtube music videos which would fit this music. The system analyzes the input music to find its genre (pop, rock, ...) and finds segmented MVs with the same genre in the database. Then, a K-Means clustering is done to group video segments by color histogram, meaning segments of MVs having the same global distribution of colors. A few clusters are randomly selected, then are assembled around music boundaries, which are moments where a significant change in the music occurs (for instance, transitioning from verse to chorus). This way, when the music changes, the video color mood changes as well. This work aims at generating high-quality realistic MVs, which could be mistaken for man-made MVs. By asking users to identify, in a batch of music videos containing professional MVs, amateur-made MVs and generated MVs by our algorithm, we show that our algorithm gives satisfying results, as 45% of generated videos are mistaken for professional MVs and 21.6% are mistaken for amateur-made MVs.

Introduction

Generating high-quality video from scratch remains a significant challenge in artificial intelligence. While generative models like GANs and diffusion models have achieved impressive results in static image synthesis, video generation often suffers from low resolution and a lack of long-term structural coherence. Music videos (MVs) present a unique challenge as they require a synchronized relationship between audio and visual "mood." Previous automated attempts often resulted in simplistic slideshows or used low-quality amateur footage. This research shifts the paradigm from pixel generation to structural assembly, focusing on intelligently remixing existing high-quality professional shots into a coherent new narrative that respects the structure of the input song.

Methodology

The proposed system treats music video generation as a structural assembly problem, utilizing a pipeline that ensures both visual quality and rhythmic alignment.

Database Preparation and Segmentation

The system utilizes a subset of the YouTube-8M dataset, specifically focusing on professional music videos. Each video is segmented into individual "shots"—continuous scenes without cuts—using content-aware detection. For every shot, a 768-size color histogram representing R, G, and B distributions is calculated and stored alongside duration data.

The preprocessing pipeline: separating source videos into shots and extracting color and duration features.

Audio Analysis and Assembly

The generation process involves three key steps:

  1. Genre Matching: The genre of the input audio (e.g., Pop, Rock, Hip-Hop) is identified via fingerprinting APIs (ACR Cloud/Last.fm) to ensure stylistic consistency between the music and the source footage.
  2. Boundary Detection: Instead of simple beat-matching, the system employs Ordinal Linear Discriminant Analysis (OLDA) to identify major structural shifts in the music, such as transitions between verses and choruses.
  3. Color Clustering and Selection: Within the identified genre, video segments are grouped into 90 clusters using K-Means based on their color distributions. The system selects five clusters to represent the "moods" of the new video.
Examples of diverse video segments extracted from the database for assembly.

The final assembly fills the intervals between detected music boundaries with shots from a single color cluster. When a structural boundary is reached, the system switches to a different color cluster, mimicking professional editing techniques where visual shifts signal musical transitions.

Results

The effectiveness of the algorithm was evaluated through a user study on Amazon Mechanical Turk. Participants were asked to classify a mix of professional, amateur, and AI-generated videos.

The results demonstrate a high level of perceived realism:

  • 45% of AI-generated videos were mistaken for professional productions.
  • 21.6% were mistaken for amateur (fan-made) music videos.
  • Only 32.8% of users correctly identified the videos as AI-generated.

The system achieved a high specificity of 0.79, indicating that users were unlikely to label a video as "generated" unless it displayed significant incoherence. These findings suggest that by prioritizing structural logic and high-quality source material over raw pixel generation, the system can produce results that rival human-made content.

Discussion and Future Work

A key insight of this work is that color distribution serves as an effective proxy for visual style. By organizing shots into distinct color "moods" and aligning transitions with musical boundaries, the system achieves a level of intentionality often missing in automated tools.

However, limitations remain. The system occasionally pulls unsuitable segments (like lyric videos) due to database noise. Furthermore, the lack of lip-syncing and semantic narrative coherence—where characters or objects persist across shots—remains a challenge. Future iterations could incorporate vocal-aware segment matching and object tracking to enhance the narrative flow and realism of the generated videos.

Summary updated

Abstract

A Music Video (MV) is a video aiming at visually illustrating or extending the meaning of its background music. This paper proposes a novel method to automatically generate, from an input music track, a music video made of segments of Youtube music videos which would fit this music. The system analyzes the input music to find its genre (pop, rock, ...) and finds segmented MVs with the same genre in the database. Then, a K-Means clustering is done to group video segments by color histogram, meaning segments of MVs having the same global distribution of colors. A few clusters are randomly selected, then are assembled around music boundaries, which are moments where a significant change in the music occurs (for instance, transitioning from verse to chorus). This way, when the music changes, the video color mood changes as well. This work aims at generating high-quality realistic MVs, which could be mistaken for man-made MVs. By asking users to identify, in a batch of music videos containing professional MVs, amateur-made MVs and generated MVs by our algorithm, we show that our algorithm gives satisfying results, as 45% of generated videos are mistaken for professional MVs and 21.6% are mistaken for amateur-made MVs. More information can be found in the project website: http://ml.cs.tsinghua.edu.cn/~sarah/
BibTeX
@misc{gross2019automatic,
  author = {Sarah Gross and Xingxing Wei and Jun Zhu},
  title = {{Automatic Realistic Music Video Generation from Segments of Youtube Videos}},
  year = {2019},
  eprint = {1905.12245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.12245},
}
Figure