Modeling Beats and Downbeats with a Time-Frequency Transformer

Yun-Ning Hung, Ju-Chiang Wang, Xuchen Song, Wei Tsung Lu, Minz Won

TL;DR

This paper introduces SpecTNT, a novel Transformer-based architecture for music beat and downbeat tracking. By jointly modeling spectral and temporal features within a hierarchical Transformer structure, SpecTNT achieves state-of-the-art performance in downbeat tracking and competitive results in beat tracking, outperforming traditional Temporal Convolutional Networks (TCNs).

What Problem Does This Paper Address?

Accurately identifying the beat (the underlying pulse of music) and the downbeat (the first beat of a measure) is a fundamental task in music information retrieval (MIR). While beat tracking has seen significant progress with deep learning, downbeat tracking remains a challenge due to its reliance on more complex musical context, such as harmonic changes and structural cues. Existing methods often struggle to capture these long-range dependencies and intricate musical relationships effectively.

Methods

The core innovation is SpecTNT (Spectral-Temporal Transformer in Transformer). This architecture leverages the power of Transformers in a novel way for audio analysis:

  • Time-Frequency Input: It processes music audio represented as a time-frequency spectrogram.
  • Hierarchical Transformer Blocks: Each SpecTNT block contains two levels of encoders:
  • Spectral Encoder: This lower-level encoder focuses on capturing harmonic and timbral components within individual audio frames. This is crucial for downbeat detection, as downbeats often coincide with harmonic shifts.
  • Temporal Encoder: This upper-level encoder aggregates the spectral information across time, allowing the model to attend to relevant temporal positions for beat and downbeat prediction.
  • Parameter Efficiency: The hierarchical design of SpecTNT allows for fewer parameters compared to the original Transformer, improving generalization on smaller MIR datasets.
  • SpecTNT-TCN Hybrid: To further boost performance, the authors propose combining SpecTNT with Temporal Convolutional Networks (TCNs). This hybrid model uses a two-stream architecture where SpecTNT and TCN process the audio in parallel, and their outputs are averaged. This aims to leverage the strengths of both architectures and mitigate potential overfitting issues during training.

Datasets

The models were evaluated on a combination of nine public datasets: Beatles, Ballroom, SMC, Hainsworth, Simac, HJDB, RWC-Popular, and Harmonix Set. The Harmonix Set is noted as being significantly larger, containing over 900 pop/rock songs. The GTZAN dataset was used exclusively for testing. The authors mention manually adjusting annotations for the Harmonix Set to ensure accuracy.

Key Insights and Ideas

  • Spectral Context is Key for Downbeats: The success of the spectral encoder in SpecTNT highlights the importance of analyzing harmonic and timbral features for accurately identifying downbeats, which are often linked to chord changes.
  • Hierarchical Transformers for Audio: Applying a hierarchical Transformer structure that explicitly models both spectral and temporal dimensions is a promising direction for audio processing tasks, offering a more nuanced understanding than purely temporal or spectral models.
  • Transformer's Data Hunger Mitigated: The SpecTNT architecture's parameter efficiency makes it more suitable for MIR tasks where large annotated datasets are scarce, a common challenge for Transformer models.
  • Hybrid Models Offer Complementary Strengths: Combining SpecTNT with TCNs demonstrates that different neural network architectures can be complementary, leading to improved performance by addressing different aspects of the music signal.

Key Results

  • State-of-the-Art Downbeat Tracking: SpecTNT significantly outperforms the TCN baseline in downbeat tracking across most datasets, achieving state-of-the-art results.
  • Competitive Beat Tracking: SpecTNT maintains comparable performance to TCN in beat tracking while offering improvements in downbeat tracking.
  • SpecTNT-TCN Boost: The combined SpecTNT-TCN model further enhances performance, particularly in downbeat tracking, and shows improved robustness.
  • Qualitative Improvements: SpecTNT is observed to handle expressive timing better and avoid some phase errors that TCNs might exhibit.
  • Ablation Study: A comparison with a "regular Transformer" (without the spectral encoder) on GTZAN shows that the spectral component is crucial for effective performance, especially with limited data.

Limitations and Future Work

  • Sub-optimal Issue: The authors acknowledge a potential "sub-optimal issue" where SpecTNT might overfit to beat annotations early in training, potentially hindering optimal downbeat tracking. This is particularly noted for datasets with diverse genres or less common musical styles.
  • Future Directions:
    • Mitigating the sub-optimal issue with improved training strategies.
  • Exploring unified multi-task learning frameworks that jointly model SpecTNT with other MIR tasks like music structure analysis, chord recognition, and melody extraction.

Summary updated

Abstract

Transformer is a successful deep neural network (DNN) architecture that has shown its versatility not only in natural language processing but also in music information retrieval (MIR). In this paper, we present a novel Transformer-based approach to tackle beat and downbeat tracking. This approach employs SpecTNT (Spectral- Temporal Transformer in Transformer), a variant of Transformer that models both spectral and temporal dimensions of a time-frequency input of music audio. A SpecTNT model uses a stack of blocks, where each consists of two levels of Transformer encoders. The lower-level (or spectral) encoder handles the spectral features and enables the model to pay attention to harmonic components of each frame. Since downbeats indicate bar boundaries and are often accompanied by harmonic changes, this step may help downbeat modeling. The upper-level (or temporal) encoder aggregates useful local spectral information to pay attention to beat/downbeat positions. We also propose an architecture that combines SpecTNT with a state-of- the-art model, Temporal Convolutional Networks (TCN), to further improve the performance. Extensive experiments demonstrate that our approach can significantly outperform TCN in downbeat tracking while maintaining comparable result in beat tracking.
BibTeX
@inproceedings{hung2022modeling,
  author = {Yun-Ning Hung and Ju-Chiang Wang and Xuchen Song and Wei Tsung Lu and Minz Won},
  title = {{Modeling Beats and Downbeats with a Time-Frequency Transformer}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=_y29_SpeOO},
}
Figure