Introduction
This paper presents a web-based system designed for live improvisational music performance within laptop ensembles, assisted by Recurrent Neural Networks (RNNs). By utilizing the Magenta.js library, the system creates a collaborative environment where a "conductor" establishes a harmonic framework while "performers" leverage local AI models to generate and manipulate melodies in real-time.
The development of this system addresses two primary challenges in live machine-learning-assisted music: technical latency and musical accessibility. Standard deep learning models often cause the main thread to "freeze" during sequence generation, which is disruptive for live audio. Furthermore, laptop ensembles frequently include participants with varying musical expertise. This system provides a "plug-and-play" solution that acts as a musical "safety net," allowing novices to contribute coherent ideas alongside experts.
System Architecture and Methods
The system is built on a central Node.js server that communicates with client browsers via a WebSocket API. The architecture distinguishes between two primary roles:
- The Conductor (Host): Responsible for controlling the global tempo and the overarching chord progression.
- The Performers (Clients): Each performer runs a local instance of an LSTM model (
chord-pitches-improv) from Google’s Magenta project.
To mitigate the "freeze" caused by JavaScript's single-threaded nature during 16-bar melody generation, the authors implemented staggered note sequence generation. By dynamically scheduling when each client generates its next sequence, the system ensures the entire ensemble does not drop out simultaneously. The server re-synchronizes clients to the master clock once the generation process completes.
Creative Control and Human-AI Interaction
Performers interact with the AI through a continuum of engagement:
- Reject: Ignoring the AI to play manually.
- Rely: Allowing the AI to generate all notes while the performer focuses on "sculpting" timbre and volume.
- Blend: Using AI-generated sequences as a creative springboard.
Beyond note generation, performers have high-level controls over the AI, such as "temperature" (randomness) and transposition, as well as low-level synthesizer controls (oscillators and envelopes) via Tone.js. This allows for emergent behaviors, such as transforming a melodic RNN into a "wind-chime" texture generator by adjusting temperature and envelope settings.
Results and Performance Feasibility
The system's feasibility was demonstrated through live performances involving four participants over a local WLAN. In a successful four-minute performance, the system remained musically cohesive despite the inherent technical constraints.
While the resulting music often exhibited a "jolly nursery-rhyme" aesthetic due to the AI's reliance on major triads and quantized timing, it ensured harmonic consistency across the group. The "plug-and-play" nature of the web-based tool allowed new performers to contribute effectively after minimal practice.
Limitations and Future Directions
Despite its success, the system still experiences UI and audio pauses during sequence generation. Future iterations could utilize AudioWorklets to move audio processing to a separate thread or implement automatic volume fading during these "freezes." Additionally, the system is highly sensitive to network stability, with standard public WiFi often causing synchronization issues. Future work will also explore more advanced generative models to expand the musical complexity beyond basic harmonic structures.
Summary updated