TL;DR
This paper tackles the challenging problem of separating individual melodic lines (voices) within a polyphonic musical piece, treating it as a Multi-Trajectory Tracking (MTT) problem. They introduce a novel approach using heterogeneous graph neural networks to learn note embeddings and predict links between consecutive notes, achieving state-of-the-art results on classical music datasets.
What Problem Does This Paper Address?
Music, especially complex pieces like those for piano or string quartets, often consists of multiple independent melodic lines or "voices" played simultaneously. The goal of voice separation is to disentangle these individual lines from the combined musical texture. This is crucial for various music information retrieval tasks, such as melody extraction and automatic transcription. Existing methods often rely on domain-specific heuristics or complex global optimization, which can limit their generalization and scalability. This paper aims to address these limitations by framing voice separation as a link prediction problem within a Multi-Trajectory Tracking (MTT) framework, using a data-driven approach that doesn't require explicit musical rules.
Methods
The core of the proposed method is to model a musical piece as a heterogeneous graph. Each note in the piece becomes a node, and various temporal relationships between notes (e.g., starting at the same time, one note sounding while another starts, one note following another) define different types of directed edges. A heterogeneous graph neural network (GNN) is then employed to learn rich node embeddings that capture both intra-note (intrinsic properties) and inter-note (contextual relationships) information.
These embeddings are fed into a link predictor, which is essentially a multi-layer perceptron (MLP). This predictor makes a binary decision for every potential pair of notes: should they be linked as consecutive notes within the same voice? This is framed as a greedy link prediction task. To enforce the constraint that each voice is monophonic (at most one incoming and one outgoing link per note), a novel regularization loss is introduced. This loss encourages the predicted adjacency matrix to resemble a structure suitable for linear assignment. Finally, a post-processing step using the Hungarian algorithm can be applied to further refine the predictions and ensure the linear assignment property.
Datasets
The authors trained and evaluated their model on a large dataset comprising 1136 pieces, including:
- 474 pieces from the Symbolic Multitrack Contrapuntal Music Archive (MCMA).
- 662 pieces from the KernScore project, specifically Bach Chorales, Haydn string quartets, and Mozart string quartets.
These datasets consist of quantized notes with pitch, onset, and offset information, along with ground truth voice separations provided by musical experts. The authors note that they discard information like rests, slurs, and stem directions, focusing solely on the note data.
Key Insights and Ideas
- Framing voice separation as MTT: This is a significant conceptual shift, allowing the problem to be tackled with established MTT techniques but adapted for the unique challenges of musical voices.
- Heterogeneous GNNs for musical context: The use of heterogeneous graphs with multiple edge types effectively captures the complex temporal and relational dependencies between musical notes, leading to more informative node embeddings than simpler graph structures.
- Greedy link prediction with regularization: The approach of predicting links greedily, combined with a novel regularization loss to enforce monophonic constraints, offers a scalable and effective way to model voice separation without relying on explicit musical rules. This is a key differentiator from many prior methods.
- Scalability and generalization: By avoiding domain-specific heuristics and global optimization, the model demonstrates better scalability to longer sequences and a higher number of voices, and improved generalization across different musical styles.
Key Results
The proposed model, GMTT, achieves new state-of-the-art results on the voice separation task across various classical music datasets.
- Without post-processing, GMTT performs competitively with existing methods, significantly outperforming them on the Haydn String Quartets dataset.
- With linear assignment post-processing, the model further improves precision and F1-score, demonstrating its ability to closely adhere to the MTT constraints.
- Ablation studies confirm the effectiveness of the heterogeneous graph structure, the residual gated convolution blocks, and the proposed regularization loss. The regularization loss, in particular, is shown to be crucial for achieving consistent and high performance.
Limitations and Future Work
- Unquantized MIDI: The current model operates on quantized symbolic music. The authors plan to extend it to handle unquantized MIDI data, exploring how expressive timing and intensity deviations correlate with voice information.
- Chords within voices: The current model assumes monophonic voices. Future work includes relaxing this assumption to handle pieces where a single voice might contain multiple simultaneous notes (chords).
- Raw audio separation: The authors express interest in developing a voice separation method from raw audio, which could incorporate information not present in symbolic formats and potentially blend with instrument/source separation tasks.
- Perceptual questions: The paper raises questions about how human listeners parse complex musical textures and whether more "natural" definitions of voice separation could be explored.
Summary updated