TL;DR
This paper introduces EMPI, an embodied musical interface that integrates recurrent neural networks (RNNs) for real-time musical prediction. By allowing musicians to interact with AI models that predict their physical gestures, EMPI offers a novel way to explore human-AI musical collaboration and understand how embodied interactions influence creative performance.
What Problem Does This Paper Address?
Traditional machine learning models for music often operate on symbolic representations (like MIDI) or abstract audio, detached from the physical gestures musicians use to create sound. This disconnect limits their integration into live performance. The authors aim to bridge this gap by creating an interface that allows musicians to interact with AI models that predict their physical movements, making the AI's "intentions" sonically and physically tangible. This allows for a more embodied and intuitive form of human-AI musical improvisation, moving beyond simple note generation to predicting and responding to gestural performance.
Methods
The core of the EMPI system is a Mixture Density Recurrent Neural Network (MDRNN). This architecture is chosen because it can predict continuous real-valued outputs (like lever position and time) and model complex distributions of these outputs. Specifically, the MDRNN is trained to predict the performer's next physical input action and the time at which it will occur.
The EMPI interface itself is a highly constrained system with a single dimension of continuous input (a lever) and output. This simplicity is intentional, designed to expose the affordances of embodied predictive interaction. The system provides both sonic output (synthesized audio) and physical feedback (a motorized lever) to represent the AI's predictions. The authors trained three distinct MDRNN models: one on human-sourced performance data, one on synthetic data, and one on random noise, to explore how different training data influences the AI's behavior and the performer's perception.
Datasets
- Human-sourced data: Collected from solo improvised recordings made using the EMPI interface in a "human only" mode (no AI prediction). This dataset represents unconstrained musical gestures.
- Synthetic data: Generated to mimic plausible lever motions using sine, square, and triangle wave functions. This data aims to represent structured, repetitive patterns.
- Noise data: Uniformly sampled random numbers, intended to have no recognizable behavior.
Key Insights and Ideas
- Embodied AI for Music: Moving beyond symbolic AI, EMPI demonstrates the potential of AI that understands and predicts physical gestures in music performance. This opens up new avenues for human-AI musical collaboration.
- Constrained Interfaces Reveal Affordances: The simplicity of EMPI's single-dimension input/output design effectively highlights the unique characteristics and affordances of different predictive models.
- Training Data Matters (A Lot): The source of training data significantly impacts how performers perceive the AI's responses. Human-like data leads to perceptions of greater musicality, creativity, and influence, while synthetic and noise data offer different, sometimes unexpected, interactive affordances.
- Physical Feedback is Divisive but Influential: While not universally preferred, the motorized output lever did appear to affect the duration of performances, suggesting it plays a role in how performers engage with and understand the AI's predictions.
- AI as a Creative Partner, Not Just a Mimic: The study suggests that AI models, even those trained on non-human data, can serve as sources of inspiration and encourage performers to explore new playing styles, rather than just replicating existing ones.
Key Results
- Survey Findings: Participants rated the human-sourced model as most related to their performance, most creative, and most influential. The synthetic model was rated better than the noise model.
- Interview Insights: While surveys favored the human model, interviews revealed that participants also found value in the synthetic and noise models for their unique interactive affordances and musical styles. Some participants preferred the consistency of the noise model, while others disliked its unpredictability.
- Physical Actuation Impact: The presence or absence of the servo-actuated lever did not have a significant effect on most survey questions. However, analysis of performance lengths showed that it widened the interquartile range for the human and synth models, suggesting it influenced engagement.
- Model Distinctiveness: Participants could generally distinguish between the three models (human, synthetic, noise) without explicit instruction, indicating that the models exhibited discernible characteristics.
Limitations and Future Work
- Limited Input/Output Dimension: The single-dimension input and output of EMPI, while useful for isolating effects, limits the complexity of musical expression. Future work could explore multi-dimensional interfaces.
- Fixed Training Data: The MDRNN models had limited capacity to reflect the current performance, relying heavily on their training data. The authors suggest exploring ML systems with on-line training capabilities for more responsive interactions.
- Physical Feedback Preference: The split in preference for the servo-actuated lever suggests that future designs might need to offer more control over this feature or consider its role in different performance contexts (e.g., solo vs. ensemble).
- Exploring Other Models and Datasets: The authors propose applying other RNN architectures or datasets to EMPI to investigate a wider range of musical styles and AI affordances. They also suggest exploring how to give performers more control over which ML model is active, potentially even switching between them during a performance.
Summary updated