Drum-Aware Ensemble Architecture for Improved Joint Musical Beat and Downbeat Tracking

Ching-Yu Chiu, Alvin Wen-Yu Su, Yi-Hsuan Yang

TL;DR

This paper introduces a "drum-aware" ensemble architecture that uses blind source separation to split audio into percussive and non-percussive streams. By training specialized trackers for each stream and fusing their outputs with a learnable mechanism, the system significantly improves beat and downbeat tracking accuracy across diverse genres, from drum-heavy jungle to drum-less classical piano.

What Problem Does This Paper Address?

Automatic beat and downbeat tracking—identifying the "pulse" and the first beat of every measure—is a cornerstone of music information retrieval (MIR). However, existing models often struggle with generalization. A model trained on pop music might rely too heavily on the steady "thump" of a kick drum, causing it to fail on classical music (where drums are absent) or jazz (where drums are improvisational).

The authors argue that humans track beats by adaptively shifting their attention between percussive and non-percussive elements. Current deep learning models typically process the "mixture" (the whole song) as a single input, missing the opportunity to specialize in these distinct musical components.

Methods

The researchers proposed a multi-stage ensemble architecture:

  1. Feature Separation: An input signal is passed through a source separation module (based on the OpenUnmix BLSTM architecture). This module segregates the audio into two streams: a "drum" component and a "non-drum" component.
  2. Parallel Trackers: Instead of one tracker, the system uses three parallel Bidirectional Long Short-Term Memory (BLSTM) networks. One tracks the original mixture, one tracks the separated drums, and one tracks the non-drum parts. Each produces activation likelihoods for beats and downbeats.
  3. Learnable Fusion (The "Fuser"): This is the core innovation. A separate BLSTM-based fuser aggregates the outputs from the three trackers. The authors tested two versions: DA1, which simply concatenates the final activations, and DA2, which uses the intermediate hidden states of the trackers to provide the fuser with richer context.
  4. Post-Processing: A Hidden Markov Model (HMM) is used to smooth the fuser's output and ensure the final beat sequence follows a musically plausible tempo and meter.

Datasets

The models were trained on a massive "Merged" dataset comprising ten established benchmarks (including RWC, Ballroom, GTZAN, and Beatles). To test generalization, they evaluated the system on three "unseen" datasets representing different drum-usage profiles:

  • ASAP: 520 tracks of classical piano music (0% drum presence).
  • Rock: 200 classic rock tracks (83.2% drum presence).
  • HJDB: 235 drum-heavy tracks like Hardcore and Jungle (99.2% drum presence).

Key Insights and Ideas

  • Divide and Conquer: By providing the machine with separated stems, it can develop "specialist" knowledge. The drum tracker learns the nuances of percussive onsets, while the non-drum tracker focuses on harmonic and melodic changes.
  • Adaptive Attention: The fuser acts as an attention mechanism. In a drum-less piano piece, the fuser learns to ignore the (silent) drum tracker and rely on the non-drum and mixture trackers. In a heavy electronic track, it leans into the drum tracker's precision.
  • The "Drum-Less" Paradox: Counter-intuitively, making a model "drum-aware" actually makes it better at tracking music without drums. By explicitly isolating the drum channel, the model can more confidently identify when that channel is empty and shift its strategy accordingly.

Key Results

The proposed DA2 fuser consistently outperformed the standard baseline (which uses no source separation):

  • Drum-Heavy Music (HJDB): The model saw a +7.0% relative improvement in downbeat tracking. This suggests that isolating the drums helps the model cut through the "noise" of complex breakbeats.
  • Drum-Less Music (ASAP): Even on solo piano, the drum-aware model outperformed the baseline, proving that the fusion mechanism effectively handles the absence of percussive cues.
  • Statistical Significance: One-tailed t-tests confirmed that the performance gains over the widely-used Madmom baseline were statistically significant across almost all test sets.

Limitations and Future Work

  • Computational Overhead: Running three parallel trackers plus a source separation module is computationally more expensive than a single-stream model. This might limit its use in real-time mobile applications.
  • Source Separation Quality: The system relies on the quality of the initial separation. While the authors used Spleeter for supervision, artifacts in the separation process could potentially mislead the trackers.
  • Future Architectures: The authors suggest that replacing the BLSTMs with Transformers or Temporal Convolutional Networks (TCNs) could further improve performance, particularly in capturing long-range rhythmic dependencies in complex classical pieces.

Summary updated

Abstract

This letter presents a novel system architecture that integrates blind source separation with joint beat and downbeat tracking in musical audio signals. The source separation module segregates the percussive and non-percussive components of the input signal, over which beat and downbeat tracking are performed separately and then the results are aggregated with a learnable fusion mechanism. This way, the system can adaptively determine how much the tracking result for an input signal should depend on the input's percussive or non-percussive components. Evaluation on four testing sets that feature different levels of presence of drum sounds shows that the new architecture consistently outperforms the widely-adopted baseline architecture that does not employ source separation.
BibTeX
@article{chiu2021drum,
  author = {Ching-Yu Chiu and Alvin Wen-Yu Su and Yi-Hsuan Yang},
  title = {{Drum-Aware Ensemble Architecture for Improved Joint Musical Beat and Downbeat Tracking}},
  journal = {IEEE Signal Processing Letters},
  year = {2021},
  url = {https://openreview.net/forum?id=8Gm17cxea7},
}

Loading related content...

Figure