Abstract: This paper describes a computer accompaniment system capable of providing musical accompaniment for an ensemble of performers. The system tracks the performance of each musician in the ensemble to determine current score location and tempo of the ensemble. "Missing parts" in the composition (i.e., the accompaniment) are synthesized and synchronized to the ensemble. The paper presents an overview of the component problems of automated musical accompaniment and discusses solutions and their implementation. The system has been tested with solo performers as well as ensembles having as many as three performers.
Introduction
Presented at AAAI-94, this work by Lorin Grubb and Roger B. Dannenberg represents a significant step forward in computer music. While solo accompaniment systems existed at the time, this research tackles the more complex "social" problem of musical ensembles. In an ensemble, performers are rarely perfectly in sync; they drift in tempo, miss notes, or drop out for rests. A computer accompanist must therefore listen to all participants and determine a "consensus" tempo and score position to provide a musically pleasing performance in real-time.
System Architecture
The system architecture is modular, designed to handle multiple inputs simultaneously. It is divided into four primary software components that manage the transition from raw performance data to synchronized accompaniment.
- The Matcher: Each performer is tracked by an individual matcher using a modified Dynamic Programming algorithm. It compares the sequence of performed pitches against the digital score to find the "best fit," effectively solving a version of the Longest Common Subsequence problem.
- The Estimator: This component maintains a buffer of recent score locations and timestamps. By analyzing the relationship between score-time and real-time, it calculates the performer's current tempo.
- The Voter: The core innovation of this system, the Voter aggregates estimates from all active performers. It uses a weighted average mechanism to resolve conflicts, employing a Recency Rating (RR) to prioritize active performers and a Clustering Rating (CR) to identify and down-weight outliers who may be "lost."
- The Scheduler: The Scheduler applies musical rules to the final output. It manages the computer's performance by speeding up to catch up, jumping to new positions if necessary, or pausing if the ensemble stops.
Tracking and Estimation
The tracking process involves mapping real-time events to score positions. The system must maintain a stable estimate of tempo even when human performers exhibit expressive timing or minor errors.
By decoupling tracking from performance, the authors created a flexible framework. The system effectively implements a "musical majority rule": if most performers jump ahead, the computer follows, assuming the outlier is the one in error. This balances the fundamental trade-off between reactivity (following every human nuance) and stability (avoiding erratic jumps due to mistakes).
Experimental Evaluation
The system was evaluated using real-time performances of various musical scores, including a simple canon ("Row, Row, Row Your Boat") and classical excerpts such as Handel’s Water Music. Tests involved ensembles of up to three performers using MIDI keyboards and acoustic wind instruments via pitch-to-MIDI converters.
Key results include:
- Robustness: The clustering weight successfully prevented the accompaniment from being derailed by single-performer errors.
- Efficiency: On early 90s hardware, processing a single input took only 1.4 ms, with minimal overhead for additional performers. This allowed for lag-free performance at tempos up to 150 BPM.
Conclusion and Future Work
While successful in tracking small ensembles, the authors noted several areas for future development. The current system requires separate inputs for each instrument, leaving the challenge of polyphonic tracking from a single microphone for future research. Additionally, the authors suggest that "learning through rehearsal"—where the system adapts to the specific expressive habits of an ensemble over time—would significantly enhance the musicality of the automated accompaniment.
Summary updated