AI Song Contest: Human-AI Co-Creation in Songwriting

Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex, Monica Dinculescu, Carrie J. Cai

Abstract

Machine learning is challenging the way we make music. Although research in deep generative models has dramatically improved the capability and fluency of music models, recent work has shown that it can be challenging for humans to partner with this new class of algorithms. In this paper, we present findings on what 13 musician/developer teams, a total of 61 users, needed when co-creating a song with AI, the challenges they faced, and how they leveraged and repurposed existing characteristics of AI to overcome some of these challenges. Many teams adopted modular approaches, such as independently running multiple smaller models that align with the musical building blocks of a song, before re-combining their results. As ML models are not easily steerable, teams also generated massive numbers of samples and curated them post-hoc, or used a range of strategies to direct the generation, or algorithmically ranked the samples. Ultimately, teams not only had to manage the "flare and focus" aspects of the creative process, but also juggle them with a parallel process of exploring and curating multiple ML models and outputs. These findings reflect a need to design machine learning-powered music interfaces that are more decomposable, steerable, interpretable, and adaptive, which in return will enable artists to more effectively explore how AI can extend their personal expression.

Introduction

Deep generative models have achieved remarkable success in producing coherent music and text. However, research often focuses on algorithmic performance in isolation, leaving a significant gap in understanding how humans interact with these algorithms in complex, open-ended creative tasks. Songwriting serves as a rigorous testbed for human-AI co-creation, requiring the integration of text, symbolic music, and audio. This study investigates how musicians partner with AI, the limitations they encounter, and the strategies they employ to align rigid machine learning (ML) models with their creative visions.

Methodology

This research employs a Human-Computer Interaction (HCI) probe, conducting an inductive thematic analysis on survey data from the 2020 AI Song Contest. The contest required teams to create a three-minute song using AI. The researchers surveyed 13 teams (61 participants) with diverse musical and technical backgrounds. The analysis focused on how teams automated song components, constructed AI pipelines, and integrated generated material into final compositions.

Thematic Analysis of Co-Creation

The study identified several key themes in the human-AI creative process:

  • The Death of the End-to-End Dream: Users largely avoided end-to-end models due to a lack of hierarchical structure and control. Instead, they decomposed songwriting into modular building blocks—lyrics, melody, and bassline—using specialized models for each.
  • The "Generate then Curate" Workflow: Due to limited steerability, users treated models as stochastic generators, producing massive volumes of material to find "happy accidents" through manual curation.
  • Hacking Steerability: Teams used ingenious workarounds to impose structure, such as "priming" models with melodic contours or interpolating between latent spaces to bridge song sections.
  • The "Flare and Focus" Burden: AI introduced a parallel loop of technical management—data formatting and model chaining—that often interrupted the natural creative cycle of ideation and refinement.
The modular human-AI songwriting loop, showing the interaction between the core creative process and various ML models.

Key Results

The study revealed distinct patterns in how teams utilized AI across different musical dimensions.

  • Usage Patterns: All 13 teams employed AI for lyrics and melodies, while over half utilized it for vocal synthesis. The complexity of the AI stack varied, with teams using between one and six unique model types.
  • Architectural Strategies: Three primary patterns emerged: Stitching (layering independent outputs), Pipelines (sequential model feeding), and Joint Modeling (simultaneous generation of multiple parts).
Distribution of AI usage across different musical components by team, highlighting the variety of human-AI collaboration levels.

The practical availability of tools—such as pre-trained checkpoints and processing scripts—frequently dictated the artistic direction, sometimes more than the musicians' original goals.

Discussion and Future Directions

The findings suggest a necessary paradigm shift in AI music tool design. Rather than "black-box" end-to-end systems, future research should prioritize decomposable and context-aware models. These tools should offer granular control over musical concepts like verse, chorus, and rhythm, and be integrated directly into existing Digital Audio Workstations (DAWs) to reduce the cognitive load of technical "model-wrangling." This approach will better support artists in exploring AI as an extension of their personal expression.

Summary updated

Abstract

Machine learning is challenging the way we make music. Although research in deep generative models has dramatically improved the capability and fluency of music models, recent work has shown that it can be challenging for humans to partner with this new class of algorithms. In this paper, we present findings on what 13 musician/developer teams, a total of 61 users, needed when co-creating a song with AI, the challenges they faced, and how they leveraged and repurposed existing characteristics of AI to overcome some of these challenges. Many teams adopted modular approaches, such as independently running multiple smaller models that align with the musical building blocks of a song, before re-combining their results. As ML models are not easily steerable, teams also generated massive numbers of samples and curated them post-hoc, or used a range of strategies to direct the generation, or algorithmically ranked the samples. Ultimately, teams not only had to manage the "flare and focus" aspects of the creative process, but also juggle them with a parallel process of exploring and curating multiple ML models and outputs. These findings reflect a need to design machine learning-powered music interfaces that are more decomposable, steerable, interpretable, and adaptive, which in return will enable artists to more effectively explore how AI can extend their personal expression.
BibTeX
@inproceedings{huang2020ai,
  author = {Cheng-Zhi Anna Huang and Hendrik Vincent Koops and Ed Newton-Rex and Monica Dinculescu and Carrie J. Cai},
  title = {{AI Song Contest: Human-AI Co-Creation in Songwriting}},
  booktitle = {ISMIR},
  year = {2020},
}
Figure