Modeling Expressive Timing in Non Metronomic Music

81 concepts60 papers3 proposals

Traditional Music Information Retrieval (MIR) and automated transcription systems often operate under a convenient fiction: that music unfolds on a rigid, predictable grid. In this idealized view, tempos are constant, time signatures are fixed, and notes align perfectly with discrete metronomic beats.

But human music is inherently elastic. Whether through intentional artistic choices like rubato, accelerando, and ritardando, or through the natural micro-timing fluctuations that create a musical "groove," human performances routinely deviate from the grid. As Andreas S. Weigend observed in Connectionism for Music and Audition, representing music purely as discrete symbolic events fails to capture the continuous, fluid nuances of human performance. When computational models enforce rigid temporal assumptions, they break down when faced with classical interpretations, jazz improvisations, or non-Western musical traditions.

To build systems that truly understand music as it is performed, researchers have shifted toward modeling expressive timing—moving away from global, static grids toward dynamic, localized, and flexible representations of time.

The Structural Ambiguity of Elastic Time

Why is modeling expressive timing so difficult? The core challenge lies in a fundamental mathematical ambiguity between tempo and rhythm.

Imagine a performance where the duration between two consecutive note onsets suddenly lengthens. An automated tracking system faces an identification problem: did the performer intentionally slow down their physical playing speed (rubato), or did they play a longer structural note value (such as moving from a quarter note to a half note)?

As A. Taylan Cemgil et al. demonstrate in Tempo tracking and rhythm quantization by sequential Monte Carlo, human timing deviations create a continuous blur between these two interpretations. A rigid system cannot resolve this ambiguity. For instance, Zongyu Yin shows in Measuring When a Music Generation Algorithm Copies Too Much: The Originality Report, Cardinality Score, and Symbolic Fingerprinting by Geometric Hashing that geometric point-set comparison methods are highly sensitive to temporal fluctuations. Even minor timing jitter or expressive variations in human-like MIDI data can cause these rigid similarity measures to drop to zero, failing to recognize an identical underlying musical structure simply because it was performed with feeling.

Furthermore, as David Rosenthal notes in Machine Rhythm, human performances are rarely metronomic even when musicians intend to play with a steady pulse. Because human performers introduce non-linear timing fluctuations, synchronizing or transcribing music requires dynamic local adjustments rather than uniform global time-stretching.

From Global Constraints to Local Periodicity

Early attempts to track rhythm and tempo tried to solve this by calculating a piece-wise constant tempo or relying on global voting mechanisms over an entire audio track, as described by Georgina Tryfou et al. in Tempo Estimation Based on Linear Prediction and Perceptual Modelling. Similarly, many multi-task architectures optimized for popular music assume a relatively stable global pulse, which Matthew E. P. Davies et al. note in Multi-Task Learning of Tempo and Beat: Learning One to Improve the Other limits their generalization to fluid genres like classical music.

To resolve the ambiguity of rubato, modern tracking architectures abandon global constraints in favor of local periodicity.

Rigid Global Tracking:
[ Beat 1 ] ------> [ Beat 2 ] ------> [ Beat 3 ] ------> [ Beat 4 ]  (Fails during rubato)

Local Periodicity Tracking:
[ Beat 1 ] -> [ Dynamic Window ] -> [ Beat 2 ] -> [ Dynamic Window ] -> [ Beat 3 ]

Instead of expecting a beat to occur at a strict mathematical interval calculated from the entire song, local periodicity models dynamically update their expectations based on the immediate history of the performance.

Ching-Yu Chiu et al. implemented this intuition in Local Periodicity-Based Beat Tracking for Expressive Classical Piano Music. By shifting from global temporal constraints to local expectations, their post-processing tracker allows the system to follow dramatic, non-linear tempo swings. When a pianist slows down at the end of a phrase, the model treats the change not as an error or a structural rhythm shift, but as a localized stretching of the current temporal window.

This note-by-note tracking is crucial for automatic transcription. As Lele Liu et al. demonstrate in Performance MIDI-to-score conversion by neural beat tracking, traditional notation software that relies on rigid quantization grids produces unreadable, rhythmically chaotic scores when fed human MIDI data. By using neural networks to track tempo dynamically on a note-by-note basis, the underlying musical structure can be preserved and mapped accurately to a readable score, separating the performer's expressive timing from the composer's intended notation.

Microtiming and the Anatomy of Groove

Expressive timing does not just happen on the macro level of tempo changes; it also occurs on the micro level. Microtiming refers to the systematic, small-scale temporal deviations from a metronomic grid that occur within a single measure. Rather than being random errors, these deviations are often intentional adjustments that define a genre's "groove" or "feel."

In Tracking Beats and Microtiming in Afro-Latin American Music Using Conditional Random Fields and Deep Learning, Magdalena Fuentes et al. show that in traditions like Brazilian samba and Uruguayan candombe, musicians systematically shift note onsets ahead of or behind the metronomic pulse. Standard MIR systems that assume a rigid grid misinterpret these shifts, leading to poor beat-tracking performance. By combining deep learning for feature extraction with Conditional Random Fields (CRFs) to model sequence dependencies, the authors explicitly capture these microtiming patterns as structural features rather than noise.

This challenge is compounded in historical and live ensemble recordings. Lucas Maia et al. demonstrate in SAMBASET: A Dataset of Historical Samba de Enredo Recordings for Computational Music Analysis that automated tracking algorithms perform significantly worse on historical recordings than on modern studio productions. Modern pop and electronic tracks are often constrained by studio click tracks and quantization tools, whereas older or live recordings preserve natural micro-rhythmic variations and human expression that require adaptive modeling.

Even at the level of a single musical event, time is elastic. In A Statistical View on the Expressive Timing of Piano Rolled Chords, the authors analyze how pianists play chords. While sheet music represents a chord as a set of simultaneous notes, human performers frequently roll or arpeggiate them for expressive effect. Because the notes are spread out over time, assigning a single, functionally accurate timestamp to the chord requires an adaptive understanding of the broader musical context, rather than a simple mathematical average of the note onsets.

Bridging the Score-to-Audio Alignment Gap

The necessity of modeling expressive timing creates a practical bottleneck when training machine learning models: data alignment. High-accuracy music transcription and source separation models require ground-truth labels, which are typically derived from digital musical scores (like MIDI files). However, digital scores are perfectly quantized and lack the expressive timing variations present in real-world audio recordings.

To train models without requiring thousands of hours of tedious manual alignment, researchers have developed frameworks that learn from unaligned or weakly supervised data by explicitly modeling the synchronization gap.

Training with Unaligned Supervision (e.g., NoteEM):
[ Expressive Audio ]  <-- Dynamic Alignment Loop -->  [ Quantized Score ]
         |                                                    |
         v                                                    v
  Real-World Audio                                     Idealized Notes

Ben Maman et al. address this in Unaligned Supervision for Automatic Music Transcription in The Wild with their NoteEM framework. Instead of forcing human performances onto a rigid timeline, NoteEM uses an Expectation-Maximization approach to simultaneously predict note transcriptions and dynamically align the unquantized audio with unaligned scores during training.

A similar intuition is used in score-informed source separation. When separating an audio mixture into isolated instrument tracks, models often use a digital score as a guide. However, if a performer introduces local rubato or tempo drift, the score and audio fall out of sync, causing severe acoustic artifacts.

To prevent this, Julio J. Carabias-Orti et al. introduce a note refinement process in Improving Score-Informed Source Separation for Classical Music through Note Refinement. By treating the activation matrix of a Non-negative Matrix Factorization (NMF) model as an image, they use computer vision techniques to precisely locate the actual human onsets and offsets.

By accounting for human tempo fluctuations rather than assuming perfect score-to-audio synchronization, models like those proposed by Gordon Wichern et al. in Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervision can successfully use scores as weak supervision, adjusting for the elastic reality of human performance to achieve cleaner source separation.

Key Foundational Work

Problem-solving Strategies in a Music Transcription System (1985) — contributed 1 concept elements to this direction.
Connectionism for Music and Audition (1993) — contributed 1 concept elements to this direction.
Problem-solving Strategies in a Music Transcription System (1985) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_expressive_timing_modeling_9cb17fc7,
  title = {expressive timing modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-timing-modeling/rdr_9cb17fc77c9b43b5b284b0c493017c16},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_expressive_timing_modeling_9cb17fc7_v1,
  title = {expressive timing modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-timing-modeling/rdr_9cb17fc77c9b43b5b284b0c493017c16/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake