An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks

Charles P. Martin, Jim Tørresen

TL;DR

This paper introduces an interactive system for building predictive musical instruments (NIMEs) using Mixture Density Recurrent Neural Networks (MDRNNs). By predicting continuous control data rather than just symbolic notes, their system enables richer, more nuanced interactive musical experiences and can even run on low-power devices like Raspberry Pis.

What Problem Does This Paper Address?

Creating truly novel and expressive digital musical instruments (NIMEs) often involves complex machine learning models. While deep learning has shown promise, applying it to real-time musical performance has been challenging. Existing approaches often focus on predicting symbolic music (like MIDI notes), which doesn't fully capture the continuous, gestural nature of many NIMEs. The authors aim to bridge this gap by providing a system that allows musicians and designers to easily integrate predictive capabilities into their instruments, enabling them to predict and generate continuous control data that can augment or even lead a performance. This opens up possibilities for instruments that can anticipate a performer's actions, fill in gaps, or even develop their own musical agency.

Methods

The core of the system is an Interactive Musical Prediction System (IMPS) built around a Mixture Density Recurrent Neural Network (MDRNN).

  • MDRNN: This model combines the sequential processing power of Recurrent Neural Networks (RNNs), specifically LSTMs, with Mixture Density Networks (MDNs).
  • RNNs/LSTMs: These are excellent at learning temporal dependencies in sequential data, making them suitable for understanding musical gestures over time.
  • MDNs: Instead of outputting a single prediction, an MDN outputs parameters for a Gaussian Mixture Model (GMM). This allows the model to represent multiple possible outcomes for a given input, which is crucial for creative tasks where there might be several valid next actions or notes. It predicts the mean ($\mu$), scale ($\sigma$), and mixing coefficients ($\pi$) for a GMM.
  • IMPS System: This provides a user-friendly interface for:
  • Data Collection: Automatically logs user interaction data (e.g., sensor readings) via OSC.
  • Model Training: Allows users to train MDRNN models on collected datasets.
  • Real-time Inference: Deploys trained MDRNNs to predict future control data during live performance.
  • Interaction Paradigms: IMPS offers four modes to explore predictive interaction: "No Predictions," "Filter" (predictions play as fast as possible), "Call-and-Response" (MDRNN generates signals when the user stops), and "Battle" (MDRNN generates signals regardless of user input).

Datasets

The paper discusses training and evaluating MDRNN models on datasets generated from real-world performances using their system. Specific examples include:

  • EMPI: A dataset from a 10-minute human performance on a two-lever interface.
  • LightPad:
    • A small dataset of 12,000 touch interactions (approx. 15 minutes of performance).
  • A larger dataset of 100,000 interactions (approx. 2 hours).
  • X-Touch Mini: A dataset of 12,000 interactions from a 10-minute performance controlling an additive synthesis instrument.

The paper does not introduce a new benchmark dataset but rather uses data collected from performances with their system to train and evaluate their models.

Key Insights and Ideas

  • Predicting Continuous Control Data is Key: Moving beyond symbolic music prediction to continuous gestural data unlocks richer interactive possibilities for NIMEs. This allows for more nuanced and embodied musical expression.
  • MDRNNs are Well-Suited for Creative Tasks: The ability of MDNs to model multi-modal distributions is ideal for creative processes where multiple outcomes are artistically valid. This allows the system to capture the inherent stochasticity and ambiguity in human performance.
  • Real-time Performance on Low-Power Devices is Feasible: The system's ability to run effectively on single-board computers like the Raspberry Pi significantly lowers the barrier to entry for creating sophisticated predictive NIMEs.
  • Small Models Can Be Sufficient: For smaller datasets, smaller MDRNN models (e.g., 64 LSTM units) can perform well and avoid overfitting, making training faster and more practical for rapid prototyping.
  • Sampling Temperature Matters: Adjusting "π-temperature" (for categorical distribution) and "σ-temperature" (for Gaussian variance) is crucial for generating useful and aesthetically pleasing predictions, allowing control over the smoothness and responsiveness of the generated gestures.

Key Results

  • Training Speed: Small MDRNNs can be trained reasonably quickly on a normal computer. For example, a 64-unit network on a 12K LightPad dataset took about 70 minutes for 100 epochs on a MacBook Pro.
  • Prediction Speed: Predictions are very fast, often under 10ms, even on less powerful hardware.
  • On a MacBook Pro, a 64-unit network achieved a mean prediction time of 1.85ms.
  • Even on a Raspberry Pi 3 B+, predictions were under 10ms for 64-unit networks.
  • Prediction speed is largely unaffected by the output dimension of the MDRNN, but significantly impacted by the number of LSTM units.
  • Model Performance:
    • For small datasets (12K LightPad interactions), smaller MDRNNs (64 units) performed best, avoiding overfitting.
  • For larger datasets (100K LightPad interactions), both 64- and 128-unit networks showed reasonable validation performance.
  • Feasibility: The system was successfully demonstrated on three different NIMEs: EMPI (custom, Raspberry Pi-based), Roli LightPad, and Behringer X-Touch Mini, showcasing its versatility.

Limitations and Future Work

  • Training Time vs. Rapid Prototyping: While training is faster than some deep learning tasks, it's still slower than traditional NIME tools like Wekinator. However, the authors argue it doesn't preclude rapid prototyping, especially with iterative retraining as the NIME evolves.
  • Understanding Learned Aspects: Future work could explore what specific aspects of a performer's control style are learned by the MDRNN models and how IMPS can be best integrated into a NIME design process.
  • Sampling Temperature Exploration: Further research is needed to fully explore the impact and optimal settings for the π and σ sampling temperatures to fine-tune predictive behavior.
  • Broader Application: The authors envision predictive interaction being widely applicable in Digital Audio Workstations (DAWs), synthesis environments, and physical hardware controllers, building on the success of tools like Google's Magenta project.

Summary updated

Abstract

This paper is about creating digital musical instruments where a predictive neural network model is integrated into the interactive system. Rather than predicting symbolic music (e.g., MIDI notes), we suggest that predicting future control data from the user and precise temporal information can lead to new and interesting interactive possibilities. We propose that a mixture density recurrent neural network (MDRNN) is an appropriate model for this task. The predictions can be used to fill-in control data when the user stops performing, or as a kind of filter on the user's own input. We present an interactive MDRNN prediction server that allows rapid prototyping of new NIMEs featuring predictive musical interaction by recording datasets, training MDRNN models, and experimenting with interaction modes. We illustrate our system with several example NIMEs applying this idea. Our evaluation shows that real-time predictive interaction is viable even on single-board computers and that small models are appropriate for small datasets.
BibTeX
@inproceedings{martin2019interactive,
  author = {Charles P. Martin and Jim Tørresen},
  title = {{An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks}},
  booktitle = {NIME},
  year = {2019},
  doi = {10.5281/zenodo.3672952},
}
Figure