Demonstration of Music Plus One - A Real-Time System for Automatic Orchestral Accompaniment

Christopher RaphaelAAAI 2006

TL;DR

This paper introduces "Music Plus One," a real-time system that generates a full orchestral accompaniment for a live musician playing a pre-composed piece. It achieves this by listening to the live player, predicting their musical interpretation, and then synthesizing a time-stretched orchestral audio recording to match. This moves beyond MIDI-based accompaniments to a more natural, audio-driven experience.

What Problem Does This Paper Address?

Creating convincing and responsive automatic musical accompaniment has been a long-standing challenge. Previous systems often relied on MIDI, which is well-suited for instruments like piano and percussion but struggles to capture the nuances of other acoustic instruments. Generating a truly integrated and expressive accompaniment that can adapt to a live performer's interpretation in real-time, using actual audio, is a significant hurdle. This paper tackles the problem of creating a system that can listen to a live musician, understand their performance nuances, and generate a high-quality, synchronized orchestral accompaniment using pre-recorded audio.

Methods

The "Music Plus One" system is comprised of three core modules:

  • Listen: This module uses a Hidden Markov Model (HMM) to analyze the live musician's acoustic signal and track their progress through a musical score. It estimates note onset times in real-time, with latency determined by the acoustic data's ambiguity.
  • Predict: This module employs a belief network to model musical interpretation. It combines information from the "Listen" module (estimated note onsets), the musical score, and past rehearsals. Observable variables include note onset times, while hidden variables represent unobservable factors like local tempo and rhythmic stress. During a rehearsal phase, parameters for these variables are learned using the Expectation-Maximization (EM) algorithm. In live performance, it predicts the most likely timing of future accompaniment notes.
  • Synthesize: This module takes a pre-recorded orchestral accompaniment (e.g., a "Music Minus One" recording) and warps its playback in real-time. This time-warping is guided by the live player's performance analysis from "Listen" and the learned interpretation from "Predict." A phase vocoder is used to achieve this time-stretching, preserving phase continuity for each frequency to maintain audio quality. An offline score-matching algorithm is used to create an index into the audio recording.

Datasets

Not discussed in the paper. The authors mention using "Music Minus One" recordings, which are commercially available for practice purposes, and that the system has been tested with students at the Indiana School of Music.

Key Insights and Ideas

  • Audio-based accompaniment: Moving beyond MIDI to synthesize accompaniment directly from audio recordings offers a richer and more natural sound.
  • Real-time interpretation learning: The system's ability to learn and adapt to a musician's unique interpretation through rehearsals is crucial for creating a truly collaborative experience.
  • Predictive synthesis: Anticipating future musical events based on learned patterns and current performance allows for a seamless and responsive accompaniment.
  • Phase vocoder for time-stretching: This technique is key to dynamically adjusting the pre-recorded accompaniment to match the live performer's tempo and expressive timing without introducing audible artifacts.

Key Results

The paper demonstrates a system capable of generating a complete orchestral accompaniment that follows a live player in real-time. It has been used with musicians in traditional classical music settings and offers potential for composers working with acoustic instruments and electronica, enabling complex rhythmic coordination and fast passages. Specific quantitative results (e.g., accuracy metrics, latency figures) are not detailed in this demonstration paper.

Limitations and Future Work

The authors acknowledge that the system's latency is limited by the ambiguity of the acoustic data. While the system has been demonstrated with traditional classical music, its potential for composers writing for acoustic instruments and electronica is highlighted as an interesting future direction, particularly for its ability to provide "virtually unlimited technical facility." Further details on limitations and specific future research directions are not extensively elaborated upon in this demonstration paper.

Summary updated

BibTeX
@inproceedings{raphael2006demonstration,
  author = {Christopher Raphael},
  title = {{Demonstration of Music Plus One - A Real-Time System for Automatic Orchestral Accompaniment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure