Abstract
Large-scale text-to-music generation models have significantly enhanced music creation capabilities, offering unprecedented creative freedom. However, their ability to collaborate effectively with human musicians remains limited. In this paper, we propose a framework to describe the musical interaction process, which includes expression, interpretation, and execution of controls. Following this framework, we argue that the primary gap between existing text-to-music models and musicians lies in the interpretation stage, where models lack the ability to interpret controls from musicians. We also propose two strategies to address this gap and call on the music information retrieval community to tackle the interpretation challenge to improve human-AI musical collaboration.
Introduction
While current text-to-music models excel at generating high-quality audio, they often fail as collaborative partners. A human drummer understands that a request to make a sound "babier" implies softer transients and a muffled tone. In contrast, current AI models are often "literal-minded," either requiring highly technical inputs or relying on simple keyword matching that fails to grasp the nuance and context of human musical communication. We identify this discrepancy as the Interpretation Gap.
The Musical Interaction Framework
To analyze this problem, we introduce a formal structure to describe the musical interaction process, divided into three distinct stages:
- Expression: The user maps their internal intent to a control signal, such as a text prompt, vocalization, or gesture.
- Interpretation: The receiver (human or AI) translates that signal into a set of actionable musical parameters.
- Execution: The receiver generates the actual musical content based on those parameters.
In human-to-human collaboration, the interpretation stage is robust and active. Musicians use abstract metaphors and non-verbal cues that their peers intuitively decode. In human-AI interaction, this stage is often missing or oversimplified, forcing the user to provide precise technical controls that the model can "execute" without truly "interpreting."
Defining the Interpretation Gap
The core issue is that while research has focused heavily on Execution—the ability of a model to follow precise signals like chord progressions—it has largely ignored how humans naturally communicate musical ideas.
As shown in the visualization above, a human collaborator can interpret a vague request like "sing with more feelings" into specific dynamics. An AI model, however, may struggle to understand the control entirely, leading to a breakdown in the collaborative flow. This "literal-mindedness" prevents AI from moving beyond keyword-level matching to intent-level understanding.
Strategies for Bridging the Gap
We propose two primary strategies to develop a more robust interpretation layer in music models:
- Direct Learning: Training models on datasets that capture human-to-human musical communication, including audio-visual data of gestures and dance, as well as textual transcripts of studio sessions where musicians describe their intent.
- LLM-based Interpretation: Leveraging the reasoning capabilities of Large Language Models (LLMs) to act as a "translator." An LLM can decompose abstract, ambiguous user prompts into the specific technical controls (e.g., reverb settings, EQ cuts, or rhythmic density) that generation models can execute. However, this requires grounding LLMs in domain-specific musical knowledge to ensure they understand how "spacious" translates to audio parameters.
Analysis of the Current Landscape
A meta-analysis of existing literature reveals a bifurcated control landscape. Integrated models like MusicGen are capable of general text descriptions but struggle with multi-step instructions or linguistic nuances like negation (e.g., "music without drums"). Conversely, control enhancement modules like Music ControlNet offer excellent execution of specific tracks but shift the burden of interpretation back to the human user, who must provide the precise melodic or rhythmic signals themselves.
Conclusion and Future Work
The primary hurdle in closing the interpretation gap is the lack of comprehensive multimodal data that captures the full chain of intent, interpretation, and execution. We call on the Music Information Retrieval (MIR) community to pivot toward the "messy" reality of human collaboration. Future work should focus on creating synthetic datasets for training, integrating deep music theory into LLMs, and moving toward multi-round conversational music editing to truly realize the potential of AI as a creative partner.
Summary updated