A Laptop Ensemble Performance System using Recurrent Neural Networks

Rohan Proctor, Charles Patrick Martin

Introduction

This paper presents a web-based system designed for live improvisational music performance within laptop ensembles, assisted by Recurrent Neural Networks (RNNs). By utilizing the Magenta.js library, the system creates a collaborative environment where a "conductor" establishes a harmonic framework while "performers" leverage local AI models to generate and manipulate melodies in real-time.

The development of this system addresses two primary challenges in live machine-learning-assisted music: technical latency and musical accessibility. Standard deep learning models often cause the main thread to "freeze" during sequence generation, which is disruptive for live audio. Furthermore, laptop ensembles frequently include participants with varying musical expertise. This system provides a "plug-and-play" solution that acts as a musical "safety net," allowing novices to contribute coherent ideas alongside experts.

System Architecture and Methods

The system is built on a central Node.js server that communicates with client browsers via a WebSocket API. The architecture distinguishes between two primary roles:

  • The Conductor (Host): Responsible for controlling the global tempo and the overarching chord progression.
  • The Performers (Clients): Each performer runs a local instance of an LSTM model (chord-pitches-improv) from Google’s Magenta project.
The web-based interface for the Conductor (top) and Player (bottom), showing controls for tempo, chord progressions, and AI parameters.

To mitigate the "freeze" caused by JavaScript's single-threaded nature during 16-bar melody generation, the authors implemented staggered note sequence generation. By dynamically scheduling when each client generates its next sequence, the system ensures the entire ensemble does not drop out simultaneously. The server re-synchronizes clients to the master clock once the generation process completes.

Creative Control and Human-AI Interaction

Performers interact with the AI through a continuum of engagement:

  1. Reject: Ignoring the AI to play manually.
  2. Rely: Allowing the AI to generate all notes while the performer focuses on "sculpting" timbre and volume.
  3. Blend: Using AI-generated sequences as a creative springboard.

Beyond note generation, performers have high-level controls over the AI, such as "temperature" (randomness) and transposition, as well as low-level synthesizer controls (oscillators and envelopes) via Tone.js. This allows for emergent behaviors, such as transforming a melodic RNN into a "wind-chime" texture generator by adjusting temperature and envelope settings.

Results and Performance Feasibility

The system's feasibility was demonstrated through live performances involving four participants over a local WLAN. In a successful four-minute performance, the system remained musically cohesive despite the inherent technical constraints.

A live performance of the laptop ensemble, demonstrating the collaborative use of the RNN-based system.

While the resulting music often exhibited a "jolly nursery-rhyme" aesthetic due to the AI's reliance on major triads and quantized timing, it ensured harmonic consistency across the group. The "plug-and-play" nature of the web-based tool allowed new performers to contribute effectively after minimal practice.

Limitations and Future Directions

Despite its success, the system still experiences UI and audio pauses during sequence generation. Future iterations could utilize AudioWorklets to move audio processing to a separate thread or implement automatic volume fading during these "freezes." Additionally, the system is highly sensitive to network stability, with standard public WiFi often causing synchronization issues. Future work will also explore more advanced generative models to expand the musical complexity beyond basic harmonic structures.

Summary updated

Abstract

The popularity of applying machine learning techniques in musical domains has created an inherent availability of freely accessible pre-trained neural network (NN) models ready for use in creative applications. This work outlines the implementation of one such application in the form of an assistance tool designed for live improvisational performances by laptop ensembles. The primary intention was to leverage off-the-shelf pre-trained NN models as a basis for assisting individual performers either as musical novices looking to engage with more experienced performers or as a tool to expand musical possibilities through new forms of creative expression. The system expands upon a variety of ideas found in different research areas including new interfaces for musical expression, generative music and group performance to produce a networked performance solution served via a web-browser interface. The final implementation of the system offers performers a mixture of high and low-level controls to influence the shape of sequences of notes output by locally run NN models in real time, also allowing performers to define their level of engagement with the assisting generative models. Two test performances were played, with the system shown to feasibly support four performers over a four minute piece while producing musically cohesive and engaging music. Iterations on the design of the system exposed technical constraints on the use of a JavaScript environment for generative models in a live music context, largely derived from inescapable processing overheads.
BibTeX
@inproceedings{proctor2020laptop,
  author = {Rohan Proctor and Charles Patrick Martin},
  title = {{A Laptop Ensemble Performance System using Recurrent Neural Networks}},
  booktitle = {NIME},
  year = {2020},
  url = {https://openreview.net/forum?id=4pQPBvHamG},
}
Figure