TL;DR
This paper introduces MIDI-GPT, a controllable generative model for multitrack music composition that addresses the limitations of existing systems in terms of usability and controllability for real-world co-creative workflows. By proposing a novel tokenization strategy and incorporating attribute controls, MIDI-GPT enables users to guide music generation with fine-grained control over musical elements, leading to more original and stylistically coherent outputs.
What Problem Does This Paper Address?
The field of AI-powered music composition has seen significant advancements, but many generative models struggle with practical usability in real-world co-creative scenarios. Existing systems often lack fine-grained control, making it difficult for musicians to integrate AI-generated content into their existing workflows or to feel ownership over the output. This can lead to concerns about AI misuse and a reluctance to adopt these tools. The paper aims to bridge this gap by developing a controllable, steerable, and expressive generative model that can assist musicians rather than replace them, focusing on features like infilling, attribute control, and stylistic similarity.
Methods
The core of MIDI-GPT lies in its novel approach to representing multitrack music. Instead of interleaving events from different tracks, it concatenates tracks into a single sequence, allowing for a more structured representation. This "Multi-Track representation" is then further enhanced with a "Bar-Fill representation" specifically designed for infilling tasks, where sections of music can be replaced or completed.
Key methodological contributions include:
- Novel Tokenization: A time-ordered sequence of musical events per track, concatenated across tracks, and then nested within a piece. This differs from interleaved representations and allows for better track-level conditioning.
- Bar-Fill Representation: A specialized tokenization for infilling, where bars to be predicted are marked with a
FILL_INtoken, enabling precise control over which sections of music are generated. - Attribute Controls: The system supports conditioning generation on various musical attributes, including instrument type, musical style, note density, polyphony level, and note duration. These controls are implemented by inserting special
CONTROLtokens before the musical material. - Expressiveness Enhancement: MIDI-GPT incorporates tokens for velocity and microtiming (
DELTAtokens) to capture more nuanced expressive qualities of musical performances, leveraging the expressive data present in datasets like GigaMIDI. - Rule-Based Sampling: To ensure syntactically valid outputs, the system employs rule-based masking to prevent impossible token sequences (e.g., a
BAR_ENDtoken before note durations are determined).
Datasets
- GigaMIDI: The primary dataset used for training and evaluation. This dataset is noted for its large scale and inclusion of expressive musical performances.
- MetaMIDI: A dataset that builds upon GigaMIDI, providing features for expressive music performance detection.
Key Insights and Ideas
- Decoupling Track Information: Separating track information from note event tokens (like
NOTE_ON,NOTE_OFF,TIME_DELTA) is crucial for accommodating a large number of instruments (128 General MIDI instruments) and an arbitrary number of tracks, unlike previous models that had inherent limitations. - Infilling is More Complex than Inpainting: The paper highlights that infilling in music, especially multitrack music, is more challenging than image inpainting due to the unknown number of tokens to be generated and the interleaved nature of traditional representations. Their Bar-Fill representation directly addresses this.
- Attribute Controls are Effective: The experiments demonstrate that attribute controls for note density and note duration are effective in steering generation, though polyphony control is noted as being less effective, likely due to the complexity of tracking simultaneous notes.
- Originality Increases with Generation Length: As the length of generated musical segments increases (e.g., 4 or more bars), the likelihood of MIDI-GPT duplicating training data significantly decreases, indicating a capacity for generating novel variations.
- Style Similarity is Maintained: The model generates music that is stylistically similar to the training data, as measured by StyleRank, suggesting it produces well-formed and coherent musical pieces.
Key Results
- Originality: For 4-bar excerpts, the percentage of trials where MIDI-GPT duplicates training data decreases significantly as the number of bars increases. For 4 or more bars, the model reliably produces original variations.
- Stylistic Similarity: MIDI-GPT generates music that is stylistically similar to the training dataset, as evaluated by StyleRank. The model's output is comparable to the ground truth data, especially when generating longer segments.
- Attribute Control Effectiveness:
- Note Density: The control is highly effective, with the majority of generated notes falling within the requested density range.
- Note Duration: The control is also quite effective, with at least 70% of note durations falling within the specified range in most cases.
- Polyphony Level: This control is less effective, with median outcomes often falling below the 70% threshold, attributed to the difficulty of precisely controlling simultaneous note events.
- Real-World Applications: MIDI-GPT is being integrated into synthesizers (OP-Z), game music composition software, digital audio workstations (Cubase), and has been used in artistic projects and AI Song contests.
Limitations and Future Work
- Limitations:
- Polyphony Control: The effectiveness of polyphony level control is noted as being lower compared to note density and duration, likely due to the inherent complexity of tracking simultaneous notes.
- Dataset Bias: The model's musical style is limited by the MIDI notation and available online data, potentially under-representing non-Western styles or styles not representable in MIDI.
- Legal Status: The legal status of AI-generated content and models like MIDI-GPT is still unclear.
- Future Work:
- Real-time Generation: Optimizing the model for real-time generation in musical agents.
- Larger Models and Attention: Training larger models to expand the attention window, allowing the model to attend to longer musical structures.
- Expanded Attribute Controls: Developing and integrating more attribute controls.
- Continued Integration: Further integration of MIDI-GPT into real-world products and practices.
- Addressing Dataset Bias: Further work is needed to quantify and address biases in the training data.
Summary updated