TL;DR
This paper introduces a novel approach for simultaneously identifying multiple fundamental frequencies (F0s) and tracking them over time in polyphonic music. By employing a high-order Hidden Markov Model (HMM), the system effectively handles missing F0 data and improves the accuracy of source stream estimation, outperforming existing frame-based methods.
What Problem Does This Paper Address?
Analyzing polyphonic music, where multiple instruments play simultaneously, presents a significant challenge in accurately identifying and tracking the fundamental frequencies (F0s) of each individual sound source. This task, known as multiple-F0 tracking, is crucial for applications like music information retrieval, source separation, and automatic music transcription. A key difficulty lies in inferring the number of concurrent sound sources (polyphony) and then associating the detected F0s with specific instruments or notes over time. Existing methods often struggle with fragmented F0 detections or accurately estimating the number of sources, especially in complex musical passages.
Methods
The proposed system tackles multiple-F0 tracking in two main stages: frame-level polyphony inference and candidate trajectory tracking.
- Frame-Level Polyphony Inference: This stage, building on prior work, first estimates the instantaneous spectrum of the music signal. It then distinguishes between harmonic partials (belonging to musical notes) and noise peaks. A scoring function, based on spectral match, smoothness, and amplitude evolution, is used to evaluate hypothetical combinations of F0 candidates. An algorithm progressively increases the assumed number of sources (polyphony) and selects the most plausible F0 estimates for that frame. This process also incorporates criteria for spectral smoothness, ensuring that the estimated F0s correspond to well-behaved harmonic sources.
- High-Order HMM for Candidate Tracking: To address the issue of fragmented F0 detections across frames, the paper introduces a novel tracking mechanism. It models each musical note using a simplified HMM with two states: "attack" (onset of a note) and "sustain" (the note continues). A high-order HMM (specifically, order 2) is employed, meaning the model considers dependencies on the previous two frames. This allows the system to "jump" over missing F0 candidates and reconnect broken trajectory segments. A forward-backward dynamic programming approach is used to propagate connection weights and identify the most likely paths (trajectories) through a trellis structure.
- Source Stream Estimation via Pruning: Once candidate trajectories are established, the final step is to prune them to estimate the actual source streams. This is achieved by maximizing a global inference likelihood, which considers the observed polyphony over time. Candidate trajectories are iteratively removed based on an "accordance ratio" (how well they match intermediate F0 estimates), aiming to find the set of trajectories that best explains the observed musical structure.
Datasets
The paper mentions using a specially constructed polyphonic music database for evaluation. This database was created by rendering synthesized polyphonic music from MIDI files and instrument sound samples. The authors highlight that this method allows for verifiable ground truth, as MIDI tracks are split to ensure notes don't overlap after rendering, enabling reliable single-F0 estimation for ground truth. They also mention using instrument samples from the RWC Musical Instrument Sound Database for training spectral smoothness thresholds.
Key Insights and Ideas
- Bridging the Gap Between Frames: The core innovation is the use of a high-order HMM to connect F0 information across frames. This elegantly handles the inherent fragmentation of F0 detection algorithms, which often produce intermittent results.
- The Power of "Jumping" Trajectories: By allowing the HMM to look back two frames, the system can effectively infer the presence of a note even if its F0 is not detected in every single frame. This is a significant improvement over frame-by-frame analysis.
- Pruning for Source Stream Coherence: The final pruning stage, guided by a maximum likelihood criterion based on the overall polyphony, is crucial for refining the candidate trajectories into meaningful source streams. This step ensures that the estimated streams are musically plausible and consistent with the detected number of instruments.
- Generic Architecture: The proposed tracking mechanism is designed to be plug-and-play. It can be integrated with various frame-based F0 estimation systems, making it a versatile tool for improving existing polyphonic music analysis pipelines.
Key Results
The proposed tracking system demonstrated significant improvements in accuracy compared to frame-based estimation systems. When evaluated on a test set of synthesized polyphonic music, the tracking system achieved an overall accuracy rate of 69.79%. This outperformed the MIREX'07 version (56.56%) and a previous thesis version (64.75%) that lacked the tracking mechanism. The improvements were particularly notable for higher polyphonies (more than 3 simultaneous sources).
Limitations and Future Work
The authors acknowledge several areas for future improvement:
- Precise Onset Detection: The current system relies on a fixed probability for the "attack" state, which might not accurately capture transient or onset features, potentially leading to incorrect trajectory initializations.
- Handling Octave Streams: The system could be enhanced to better verify and handle notes that are harmonically related, such as those in octave relationships.
- Allowing Trajectory Intersection: Currently, trajectories are treated as unique. The authors suggest allowing nodes to be shared across different paths in the backward tracking stage to better handle situations where source streams might overlap or intersect (e.g., with vocal performances).
- Improved Polyphony Inference: The paper suggests that using a polyphony-dependent inference likelihood, rather than a frame-independent one, could lead to more robust results.
- Rearranging Processing Modules: The authors propose that reordering the processing modules, for instance, performing trajectory tracking before joint evaluation, might further enhance performance.
Summary updated