The ACCompanion: Combining Reactivity, Robustness, and Musical Expressivity in an Automatic Piano Accompanist

Carlos Cancino Chacón, Silvan Peter, Patricia Hu, Emmanouil Karystinaios, Florian Henkel, Francesco Foscarin, Gerhard WidmerIJCAI 2023 · DOI

TL;DR

This paper introduces the ACCompanion, an AI system that acts as a real-time piano accompanist. It synchronizes with a human soloist, adapting its tempo, dynamics, and articulation to match the performer's style, aiming for a more natural and expressive musical collaboration. The system is designed to be robust to performance errors and is made publicly available.

What Problem Does This Paper Address?

Creating an AI that can convincingly accompany a human musician in real-time is a significant challenge. Existing systems often focus on basic synchronization (keeping time) or only adapt a limited set of expressive parameters like tempo. The authors aim to build an accompanist that can not only keep pace but also capture and respond to the nuanced expressivity of a human performer – their tempo variations, dynamic shifts, and articulation choices – to create a more engaging and natural musical duet. This involves bridging the gap between symbolic music representation and real-time, expressive performance generation, while also ensuring the system is reliable and doesn't falter due to minor human performance errors.

Methods

The ACCompanion is built around two core modules:

  • Score Follower: This module's primary job is to track the human soloist's performance and align it with the musical score. They employ two approaches:
  • HMM-based: A Hidden Markov Model combined with a Kalman filter, which uses the score and performed MIDI notes (pitch, timing) to infer the current position in the score.
  • OLTW (Online Time Warping): A dynamic programming algorithm that aligns incoming performance data to a reference performance (ideally by the same musician, but pre-recorded). This approach is noted for its robustness, especially with complex musical passages.
  • Accompanist: This module takes the aligned score and performance data and generates an expressive accompaniment. It does this by:
  • Encoding Expressive Parameters: Extracting tempo (beat period), dynamics (MIDI velocity), and articulation (note duration relative to tempo) from the soloist's performance.
  • Tempo Models: Crucially, they investigate several tempo models inspired by cognitive research on sensorimotor synchronization. These models aim to predict the next tempo and onset, going beyond simple reactive adjustments to incorporate anticipatory elements. They tested variants like linear synchronization (L), linear tempo expectation (LTE), and a joint adaptation-anticipation model (JADAM), comparing them against simpler reactive (R), moving average (MA), and Kalman filter (K) models.
  • Decoding: Using the extracted expressive parameters to generate the accompaniment's MIDI performance.

The system also includes a MIDI Handler for flexible input/output routing.

Datasets

The paper mentions using two datasets for evaluation:

  • Magaloff Dataset: Contains solo piano performances recorded on computer-controlled Bösendorfer grand pianos, aligned to their scores.
  • Zeilinger Dataset: Similar to Magaloff, also containing aligned solo piano performances.

They artificially generated variations of these performances by adding Gaussian noise to note onsets and offsets to simulate variations in human playing. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Beyond Synchronization: The Expressive Partnership: The core insight is that true musical collaboration requires more than just keeping time. The AI needs to understand and adapt to the expressive intent of the human player, mirroring their nuances in tempo, dynamics, and articulation.
  • Cognitive Models for Tempo: Applying models from sensorimotor synchronization research to tempo prediction offers a more human-like and potentially more musically pleasing way for the AI to manage tempo, moving beyond purely reactive adjustments.
  • Robustness is Key: The OLTW score follower, by leveraging reference performances, demonstrates a more robust way to handle the inevitable imperfections and variations in human playing, which is crucial for a reliable accompanist.
  • Trust and Communication are Missing: A significant qualitative insight is that even with advanced technical capabilities, the AI lacks the "cognitive-emotional alignment" and "trust" that underpins human musical duets. The system's reactive nature can sometimes lead to unhelpful feedback loops, highlighting the need for better modeling of human-machine interaction dynamics.

Key Results

  • Score Following Accuracy: The OLTW score follower significantly outperformed the HMM-based follower, especially on complex pieces, achieving median absolute asynchrony as low as 60.6ms compared to HMM's 645.7ms. OLTW also showed better consistency across different pieces.
  • Tempo Model Performance: The LTE tempo model demonstrated the best performance in terms of average absolute errors for both tempo and onset prediction (23.3ms and 63.3ms/beat, respectively), likely due to its ability to leverage reference performance information.
  • Qualitative Feedback: Live demonstrations and musician feedback highlighted the system's ability to provide expressive accompaniment and adapt to playing styles. However, musicians also reported a lack of natural communication and trust, emphasizing the need for the system to understand when to lead and when to follow, and to provide clearer cues.

Limitations and Future Work

  • Lack of True Understanding: The authors acknowledge that the system, while technically impressive, lacks genuine musical understanding or the ability to perceive music like a human. This leads to limitations in collaborative expressivity and the "trust" aspect of musical partnerships.
  • Modeling the Feedback Loop: A major future direction is to explicitly model the feedback loop between the human and the AI, enabling the system to better discern when to be reactive versus proactive.
  • Visual Cues: Introducing visual feedback for the human performer (e.g., on tempo) and potentially for the system (e.g., from body sensors or cameras) is suggested to improve synchronization and communication.
  • End-to-End Accompanist: Developing a single, end-to-end model that generates expressive accompaniment directly from the soloist's performance, rather than through separate score following and accompaniment modules, is another avenue for future research.
  • Systematic User Evaluation: The authors plan more extensive user studies to evaluate the system in various configurations.

Summary updated

BibTeX
@inproceedings{chacon2023accompanion,
  author = {Carlos Cancino Chacón and Silvan Peter and Patricia Hu and Emmanouil Karystinaios and Florian Henkel and Francesco Foscarin and Gerhard Widmer},
  title = {{The ACCompanion: Combining Reactivity, Robustness, and Musical Expressivity in an Automatic Piano Accompanist}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/641},
}
Figure