TL;DR
This paper argues that machine learning research in music creation should move beyond traditional quantitative metrics and focus on real-world impact and collaboration with practitioners. Through a case study involving a public concert of AI-generated music, the authors demonstrate how this approach can lead to more meaningful research and novel musical outcomes.
What Problem Does This Paper Address?
The paper tackles a critical question in applied machine learning: how do we ensure that research is not just technically sound but also genuinely useful and impactful for its intended users? Specifically, in the domain of music creation, the authors observe that much research focuses on model performance metrics (like sequence likelihoods) or qualitative listening tests, often using datasets that are only "proxies" for real-world scenarios. This disconnect means that the true value and impact of these technologies on actual musicians and composers are rarely assessed. The paper aims to bridge this gap by explicitly involving practitioners and evaluating the machine learning applications based on their real-world usefulness and their ability to inform future research.
Methods
The core methodology revolves around a practical, collaborative approach to applying machine learning in music creation. The authors employed several machine learning models, including:
- folk-rnn: A Long Short-Term Memory (LSTM) network trained on ABC notation transcriptions of traditional Irish music. This model was used for generating new musical sequences.
- DeepBach: An LSTM-based statistical model trained on Bach chorales, capable of generating four-part chorales or harmonizing melodies.
- Morpheus: A system that generates new pieces by morphing existing template pieces, preserving long-term structure through tonal tension and recurrent pattern analysis.
- MusAIcians (Ed SheerAI vs XenAkIs vs Aldele): AI agents trained on specific artists' music to generate new chord sequences and melodies.
The key innovation lies not just in the models themselves but in how they were used: in collaboration with musicians and composers, culminating in a public concert. The research then reflects on this entire experience to derive insights.
Datasets
- ABC Notation Transcriptions: Over 23,000 ABC transcriptions of traditional tunes, primarily from Ireland and the UK, sourced from the online repository http://thesession.org.
- J.S. Bach Chorales: A dataset of 389 Bach chorales used to train the DeepBach model.
- Artist-Specific Audio Corpora: Audio files from Ed Sheeran, Iannis Xenakis, and Adele were used to train the MusAIcians models.
- Classical Music Pieces: Selections from J.S. Bach's "A Little Notebook for Anna Magdalena" and Kabalevsky's "30 Children's Pieces" and "24 Children's Pieces" were used as templates for the Morpheus system.
Key Insights and Ideas
- "Machine Learning Research That Matters": The paper strongly advocates for Wagstaff's principles: measuring concrete impact with practitioners and using those results to improve the ML application and the field in general.
- Beyond "Proxies": Traditional evaluation metrics (like sequence likelihood) are insufficient. Real-world usefulness and impact on practitioners are the true measures of success.
- Collaboration as a Creative Catalyst: Working with AI models, even those with limitations, can push composers out of their comfort zones, inspire new ideas, and lead to unique musical outcomes that wouldn't have been possible otherwise.
- The "Musical Other": AI can act as a creative partner, offering novel material and perspectives that challenge and enrich the composer's own voice.
- Context is Crucial: Music data is not independent of its context and function. Researchers must consider the ethical implications, especially when working with traditional music, to avoid trivializing cultural heritage.
- Calibrating Systems to Users: Future ML systems for music creation should be designed to be adaptable and responsive to individual user needs, rather than expecting them to work perfectly "out of the box."
Key Results
- Successful Concert and Audience Engagement: A public concert featuring music composed by or with the described AI systems was held, and audience feedback was collected.
- Diverse Musical Outputs: The project resulted in a variety of musical pieces, showcasing different applications of machine learning, from generating entire chorales to co-creating folk tunes and morphing classical pieces.
- Qualitative Insights from Practitioners: Composers and musicians reported that the AI systems provided inspiration, helped them break habits, and led to compositions they wouldn't have imagined otherwise. They also highlighted the importance of the human role in shaping and interpreting the AI's output.
- Audience Perceptions: Audience members expressed excitement about AI-human collaborations in music, with some noting a shift in their listening habits when aware of the AI's involvement, focusing more on structure, consistency, or potential "imperfections."
Limitations and Future Work
- Model Limitations: The paper acknowledges limitations in the models themselves, such as folk-rnn's difficulty in generalizing beyond its training data and Morpheus's lack of formal cadence concepts, leading to surprising endings.
- Human Editing Debate: While some composers preferred minimal editing to showcase the AI's output, others found it necessary to achieve their artistic vision. The degree of human intervention remains a point of discussion.
- Future Work: The authors suggest several avenues for future research:
- Integrating statistical models (like recursive neural networks) into systems like Morpheus to improve note-to-note transitions and formalize concepts like cadences.
- Developing more sophisticated automatic transcription technologies to improve the quality of audio analysis for AI training.
- Designing systems that can be "calibrated" to human users, adapting to their specific needs and preferences.
- Further exploring the ethical dimensions of AI in music creation, particularly concerning authenticity and cultural heritage.
Summary updated