Expressive Performance Modeling for Neural Audio Synthesis

363 concepts234 papers2 proposals

When a human musician reads a sheet music score, they do not play it like a computer executing code. A standard MIDI file or digital score is a rigid, discrete representation: it specifies exactly what notes to play and when to play them on a perfect grid. However, if you play this raw symbolic data back through a traditional synthesizer, the result sounds flat, mechanical, and lifeless.

Human performances are defined by micro-timing deviations, continuous dynamic shifts (velocity), vibrato, and subtle articulations. This discrepancy is known as the performance gap. To bridge it, researchers are developing expressive performance modeling techniques. Instead of treating music synthesis as a simple text-to-speech or note-to-frequency mapping, this field focuses on teaching neural networks to interpret the unwritten nuances of a musical score and render them into realistic, human-like audio.

Capturing Musical Context via Graph Neural Networks

Why can't a standard sequence model easily predict how a note should be performed? The core challenge is that musical notes are not just linear sequences; they are highly interconnected and hierarchical. A note's expressive weight depends on its position within a chord, its role in a melodic phrase, and its relationship to surrounding structural sections.

To capture this multi-dimensional context, Dasaem Jeong et al. shifted away from flat sequence processing in Graph Neural Network for Music Score Data and Modeling Expressive Piano Performance. The authors modeled musical scores as graphs, where individual notes serve as nodes connected by specific structural edges (such as belonging to the same voice, chord, or consecutive beat).

By passing messages along these edges, a Graph Neural Network (GNN) allows each note to "look around" its musical neighborhood. The network learns how a human pianist naturally applies rubato (slowing down or speeding up) or alters dynamics based on the surrounding harmonic and melodic tension. This structural awareness is what enables the model to infer performance parameters that a purely sequential model would miss.

Intermediate Expressive Representations

Going directly from a discrete score to a raw audio waveform is an incredibly steep optimization problem. For highly expressive instruments like the violin, the sound is defined by continuous physical movements—such as the sliding between pitches (portamento) and the rapid modulation of pitch (vibrato).

To solve this, researchers introduce intermediate expressive layers that act as a bridge between discrete notation and continuous sound. In ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning, Daewoong Kim demonstrates that conditioning an audio generation model on explicit, continuous pitch bend trajectories drastically improves realism. Instead of forcing the neural audio generator to guess the micro-variations in pitch, the model is given a continuous guide track, preventing the synthesized output from sounding unnaturally static.

[Discrete Score] ---> [Expressive Trajectory Model] ---> [Continuous Pitch/Velocity] ---> [Neural Audio Synthesizer] ---> [Realistic Audio]

This concept also applies to physical mechanics. For instruments like the guitar, the sound depends heavily on how a string is struck or muted. Adarsh Kumar et al. addressed this in DadaGP: A Dataset of Tokenized GuitarPro Songs for Sequence Models by utilizing "prescriptive" tokens. Rather than just recording pitch, their tokenization scheme includes explicit performance instructions like fingerings, palm muting, and slides. By training sequence models on these physical performance gestures, the AI learns to simulate the mechanical constraints and stylistic choices of a human player.

Conditioning Audio Diffusion on Performance Signatures

When synthesizing complex, multi-instrument music, maintaining global structural control while generating rich acoustic textures is a major hurdle. Traditional score-to-audio models often suffer from "instrument drift" or produce flat, sterile timbres because they lack direct control over the recording environment and performer style.

To decouple musical structure from acoustic texture, Ben Maman et al. introduced performance conditioning in Performance Conditioning for Diffusion-Based Multi-Instrument Music Synthesis. The authors used Feature-wise Linear Modulation (FiLM) layers to inject specific "performance IDs" directly into a diffusion-based audio synthesis pipeline.

These IDs encapsulate non-notated acoustic characteristics, such as a specific performer's touch, instrument timbre, and room acoustics. Because the diffusion model is explicitly conditioned on these signatures, it doesn't have to waste capacity trying to infer the sonic environment from the notes alone. This allows the system to generate highly realistic multi-instrumental tracks with consistent textures while strictly adhering to the input score.

Cross-Modal and Vocal Performance Alignment

The necessity of modeling expressive micro-timing is perhaps most obvious in singing voice synthesis and cross-modal generation, where human perception is highly sensitive to unnatural alignments.

In vocal synthesis, a singer rarely hits a note boundary at the exact millisecond dictated by a metronome. Heiga Zen tackled this in An HMM-based singing voice synthesis system by introducing explicit "time-lag" modeling. This mechanism accounts for the natural, subtle timing differences between musical note boundaries and actual vocal onsets (such as a consonant starting slightly before the beat), which is essential for making synthesized singing sound organic.

Similarly, when generating music from visual inputs—such as a silent video of a musician playing an instrument—direct pixel-to-waveform mapping often results in unstructured noise. As shown by Chuang Gan et al. in Foley Music: Learning to Generate Music from Videos and Kun Su in Audeo: Audio Generation for a Silent Performance Video, the key to success is mapping visual gestures (like hand positions or body keypoints) to intermediate symbolic or performance representations first. By translating the physical motion of a performance into structured expressive data before rendering the audio, these systems ensure that the final sound matches both the precise timing and the physical intensity of the human performer.

Key Foundational Work

A Case Based Approach to the Generation of Musical Expression (1999) — contributed 4 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 253 papers
  2. Version 1 · 2026-10-04 · 389 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_expressive_performance_modeling_27cf5047,
  title = {expressive performance modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-performance-modeling/rdr_27cf5047da544da8ab655d56f891078f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_expressive_performance_modeling_27cf5047_v2,
  title = {expressive performance modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-performance-modeling/rdr_27cf5047da544da8ab655d56f891078f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake