Abstract
This paper introduces a hierarchical, Hidden Markov Model (HMM)-based approach to align musical scores with audio recordings. By leveraging the inherent structure of music (bars and beats) to adaptively prune the search space, the proposed method achieves higher alignment accuracy than standard Dynamic Time Warping (DTW) while significantly reducing computational complexity.
Introduction
Music-to-score alignment, or audio-to-score synchronization, is the task of determining the precise timing of musical events from a score within an audio performance. While real-time systems exist for score following, off-line systems can leverage the entire recording for higher precision. Historically, Dynamic Time Warping (DTW) has been the industry standard. However, DTW suffers from quadratic computational complexity ($O(N^2)$) and difficulties in fusing heterogeneous audio features, such as pitch and percussive onsets. This research proposes a "coarse-to-fine" search strategy using an HMM framework to address these limitations.
Methodology
The HMM Framework
The authors transition from distance-based DTW to a probabilistic HMM framework. In this model, states represent "chords" (simultaneous notes) from the score. Two primary observation models are used:
- Chroma Vectors: Representing pitch content via a histogram model of expected distributions.
- Spectral Flux: Capturing onsets through a logistic model to identify the "attack" of notes.
A key refinement is the Onset Model, which decomposes each chord state into an "attack" phase ($A$) and a "sustain" phase ($S$). This allows the system to specifically target the energy burst at the start of a note, significantly improving precision in polyphonic contexts.
Hierarchical Pruning
To overcome the efficiency drawbacks of a full-grid search, the authors introduce a hierarchical pruning strategy. Alignment occurs in three successive stages:
- Bar Level: Coarse alignment using 1s windows.
- Beat Level: Medium alignment using 200ms windows.
- Chord Level: Final high-precision alignment.
Unlike fixed-radius pruning, this method employs an adaptive likelihood threshold ($\eta$). The search space dynamically expands during periods of model uncertainty and contracts when confidence is high. This adaptive windowing, calculated via a forward-backward Viterbi variant, ensures the correct path is maintained even during repetitive or confusing musical sections.
Results
The system was evaluated using 94 pop songs from the RWC-pop database, subjected to artificial random tempo changes (40 to 240 bpm) to simulate performance variability.
- Accuracy: The "Onset" model achieved a recognition rate of 87.16% (onsets within 300ms), surpassing the best DTW baseline of 86.07%.
- Efficiency: Hierarchical pruning reduced the search space to approximately 3% of the full DTW grid.
- Performance: The pruned system completed alignment in 240 seconds, nearly twice as fast as the non-pruned version (482 seconds), with no loss in accuracy.
Conclusion
By integrating musical structure (bars and beats) directly into the search constraints, the authors demonstrate that hierarchical HMMs can outperform traditional DTW in both accuracy and speed. Future work may involve incorporating rhythmic priors (note durations) into transition probabilities and exploring more complex models at the finest level of the hierarchy, made feasible by the efficiency gains of pruning.
Summary updated