All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio

Tae‐Jun Kim, Juhan NamWaspaa 2023

TL;DR

This paper introduces an "all-in-one" deep learning model that simultaneously predicts musical beat, downbeat, segmentation boundaries, and functional structure labels. By leveraging novel "neighborhood attention" mechanisms within a transformer architecture and processing source-separated audio, the model achieves state-of-the-art performance across all four tasks on the Harmonix dataset with a remarkably compact parameter count.

What Problem Does This Paper Address?

Music is inherently hierarchical, with metrical elements like beats and bars forming the foundation, and functional elements like verses and choruses building upon them. However, existing research in Music Information Retrieval (MIR) has largely treated these hierarchical levels as separate tasks (e.g., beat tracking, segmentation, structure labeling). This siloed approach misses the potential benefits of joint learning, where information from one level could inform and improve predictions at others. The challenge lies in developing a unified model that can efficiently process the long, complex temporal sequences of music and capture dependencies across these diverse hierarchical levels.

Methods

The proposed model builds upon the transformer architecture, incorporating two key innovations:

  • Dilated Neighborhood Attention (DiNA): This mechanism models long-term temporal dependencies by using dilated convolutions. Crucially, it allows the model to capture information from distant past frames without requiring extensive zero-padding, thus reducing computational complexity, especially at the end of audio sequences.
  • 2D Neighborhood Attention (NA): This block models inter-instrument dependencies by focusing on local neighborhoods in the time-frequency domain. This differs from previous approaches that might use fixed sliding windows or single-frame attention for instrumental relationships.

The model also utilizes source-separated spectrograms as input, a technique that has shown promise in prior work. It employs a stack of transformer modules, alternating between DiNA and NA blocks, to capture both temporal and instrumental relationships. For post-processing, it uses a dynamic Bayesian network for beat and downbeat tracking, and a peak-picking method for segmentation and structure labeling.

Datasets

The primary dataset used for evaluation is the Harmonix Set [17]. This dataset is notable for containing annotations for all four tasks the paper addresses: beat, downbeat, segmentation boundaries, and functional structure labels (e.g., verse, chorus). The authors mention performing data cleaning and label merging to align with previous work. No additional datasets or data augmentation were used for training.

Key Insights and Ideas

  • Synergy in Multi-Task Learning: The ablation study strongly suggests that joint learning of beats, downbeats, and segmentation significantly benefits all three tasks. This indicates that these metrical and structural elements are indeed interdependent and can inform each other's prediction.
  • Structure Labeling's Unique Nature: While metrical and segmentation tasks benefit from joint learning, functional structure labeling appears to be less directly influenced by the other tasks. The authors hypothesize this is because structure labeling might rely more on long-term timbre textures, whereas the other tasks are more focused on short-term event detection.
  • Efficiency of Neighborhood Attention: The DiNA mechanism's ability to widen the receptive field without zero-padding is a significant computational advantage, especially for long-term dependencies in music.
  • Demixed Sources are Crucial: Removing the demixed input significantly degrades performance across all tasks, highlighting the importance of separating musical sources before analysis.

Key Results

The proposed "All-in-One" model achieves state-of-the-art performance on the Harmonix Set for all four tasks: beat tracking, downbeat tracking, segmentation, and functional structure labeling. Specifically:

  • Beat Tracking: Achieved an F1 score of 0.958.
  • Downbeat Tracking: Achieved an F1 score of 0.915.
  • Segmentation: Achieved an HR.5F score of 0.660.
  • Structure Labeling: Achieved a PWF score of 0.738.

Remarkably, the model achieves these results with a relatively small parameter count (around 300K), comparable to or smaller than many previous state-of-the-art models that focused on only one or two of these tasks. A smaller version of the model (46K parameters) also achieved state-of-the-art results in segmentation and structure labeling.

Limitations and Future Work

  • Structure Labeling Independence: The authors note that structure labeling doesn't seem to benefit as much from joint learning with the other tasks. Future work could explore how to better integrate this higher-level information.
  • Segment Length Impact: While longer segments generally led to better generalization, the authors mention the practical limitation of GPU memory for processing very long sequences. Future work could explore more efficient ways to handle extremely long audio inputs.
  • Model Size: While the model is relatively compact, further research into even more lightweight architectures could be beneficial for real-time applications.
  • Data Augmentation: The study did not utilize data augmentation, which could potentially further improve performance.

Summary updated

Abstract

Music is characterized by complex hierarchical structures. Developing a comprehensive model to capture these structures has been a significant challenge in the field of Music Information Retrieval (MIR). Prior research has mainly focused on addressing individual tasks for specific hierarchical levels, rather than providing a unified approach. In this paper, we introduce a versatile, all-in-one model that jointly performs beat and downbeat tracking as well as functional structure segmentation and labeling. The model leverages source-separated spectrograms as inputs and employs dilated neighborhood attentions to capture temporal long-term dependencies, along with non-dilated attentions for local instrumental dependencies. Consequently, the proposed model achieves state-of-the-art performance in all four tasks on the Harmonix Set while maintaining a relatively lower number of parameters compared to recent state-of-the-art models. Furthermore, our ablation study demonstrates that the concurrent learning of beats, downbeats, and segments can lead to enhanced performance, with each task mutually benefiting from the others.
BibTeX
@inproceedings{kim2023all,
  author = {Tae‐Jun Kim and Juhan Nam},
  title = {{All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio}},
  booktitle = {Waspaa},
  year = {2023},
}
Figure