Abstract
This paper presents a novel system architecture that integrates blind source separation with joint beat and downbeat tracking in musical audio signals. The source separation module segregates the percussive and non-percussive components of the input signal, over which beat and downbeat tracking are performed separately and then the results are aggregated with a learnable fusion mechanism. This way, the system can adaptively determine how much the tracking result for an input signal should depend on the input's percussive or non-percussive components. Evaluation on four testing sets that feature different levels of presence of drum sounds shows that the new architecture consistently outperforms the widely-adopted baseline architecture that does not employ source separation.
Introduction
Automatic beat and downbeat tracking—identifying the "pulse" and the first beat of every measure—is a cornerstone of music information retrieval (MIR). However, existing models often struggle with generalization. A model trained on pop music might rely too heavily on the steady "thump" of a kick drum, causing it to fail on classical music where drums are absent, or jazz where drums are improvisational.
The authors argue that humans track beats by adaptively shifting their attention between percussive and non-percussive elements. Current deep learning models typically process the "mixture" (the whole song) as a single input, missing the opportunity to specialize in these distinct musical components.
Methodology
The researchers proposed a multi-stage ensemble architecture designed to mimic human adaptive attention by processing separated audio streams.
The architecture consists of four primary stages:
- Feature Separation: An input signal is passed through a source separation module (based on the OpenUnmix BLSTM architecture). This module segregates the audio into a "drum" component and a "non-drum" component.
- Parallel Trackers: Instead of one tracker, the system uses three parallel Bidirectional Long Short-Term Memory (BLSTM) networks. One tracks the original mixture, one tracks the separated drums, and one tracks the non-drum parts. Each produces activation likelihoods for beats and downbeats.
- Learnable Fusion (The "Fuser"): This is the core innovation. A separate BLSTM-based fuser aggregates the outputs from the three trackers. The authors tested two versions: DA1, which concatenates final activations, and DA2, which utilizes intermediate hidden states for richer context.
- Post-Processing: A Hidden Markov Model (HMM) smooths the fuser's output to ensure the final beat sequence follows a musically plausible tempo and meter.
Data and Evaluation
The models were trained on a massive "Merged" dataset comprising ten established benchmarks. To test generalization, the system was evaluated on three "unseen" datasets with varying drum-usage profiles:
- ASAP: Classical piano music (0% drum presence).
- Rock: Classic rock tracks (83.2% drum presence).
- HJDB: Drum-heavy tracks like Hardcore and Jungle (99.2% drum presence).
Results and Discussion
The proposed DA2 fuser consistently outperformed the standard baseline architecture. By providing the machine with separated stems, it develops "specialist" knowledge: the drum tracker learns percussive onsets, while the non-drum tracker focuses on harmonic and melodic changes.
Key findings include:
- Improved Precision in Drum-Heavy Music: On the HJDB dataset, the model achieved a significant relative improvement in downbeat tracking. Isolating the drums helps the model cut through the "noise" of complex breakbeats.
- The "Drum-Less" Paradox: Counter-intuitively, making a model "drum-aware" improves performance on music without drums (ASAP dataset). By explicitly isolating the drum channel, the model can identify when that channel is empty and shift its strategy to rely on non-drum cues.
- Statistical Significance: One-tailed t-tests confirmed that the performance gains over the widely-used Madmom baseline were statistically significant across almost all test sets.
Conclusion
This research demonstrates that integrating source separation with beat tracking allows for a more robust and adaptive system. While the computational overhead of parallel trackers is higher, the gains in generalization across diverse musical genres are substantial. Future work may explore replacing BLSTMs with Transformers or Temporal Convolutional Networks (TCNs) to better capture long-range rhythmic dependencies.
Summary updated