Abstract
Predicting upcoming musical events is a fundamental aspect of human music perception. In computational audio, achieving this in a strictly causal, real-time setting is challenging, especially for continuous-valued features like timbre and harmony. This paper presents a framework that combines short-term predictive models with a novel long-term repetition detection algorithm. By leveraging the macro-structure of music through Self-Distance Matrices (SDM) and employing a dynamic ensemble weighting scheme, the proposed method achieves a 13% average improvement in prediction accuracy over standard linear baselines.
Introduction
When listening to music, humans subconsciously anticipate structural changes, such as the return of a chorus or a rhythmic "drop." Replicating this predictive capability computationally is vital for real-time applications, including offsetting network latency in live streaming, driving synchronized visualizations, or enabling automated accompaniment. While previous work has focused on discrete symbolic data (e.g., MIDI) or offline analysis, this research addresses the more difficult task of causal prediction on continuous audio. The goal is to predict the numerical values of future audio frames using only past information as the music unfolds.
Methodology
The study evaluates three primary predictive approaches and an ensemble method to forecast beat-synchronous Mel-frequency cepstral coefficients (MFCCs) and chroma features.
- Autoregressive Models: Multivariate autoregressive (MAR) and Linear Predictive Coding (LPC) models serve as baselines, assuming future features are linear combinations of the recent past.
- N-gram Models: Adapted from natural language processing, these models require discrete symbols. The authors use online k-means clustering to quantize continuous features into a codebook on the fly.
- Repetition Detection (SDM): This novel approach leverages musical macro-structure. By maintaining a Self-Distance Matrix (SDM), the system compares the current sequence of features with all previous sequences. When a similar past sequence is found, the model predicts that the music will continue as it did previously.
- Dynamic Ensemble Weighting: A Bayesian-inspired linear weighting scheme combines these predictors. It dynamically adjusts weights based on a running estimate of each predictor's recent error variance, favoring models that have been most accurate in the immediate past.
Experimental Setup
The evaluation was conducted using 180 mono audio tracks from The Beatles. Beat-synchronous chroma (harmony) and MFCC (timbre) features were extracted. Although the prediction task is causal, beat-onset times were determined using a non-causal "oracle" algorithm to ensure synchronization for the experiment.
Results and Discussion
Performance was assessed using Sum of Squares Error (SSE), Jensen-Shannon Divergence (JSD), and Absolute Deviation (AD), with the LPC model as the primary baseline.
The results indicate that the SDM predictor significantly outperforms a simple "copy" baseline, demonstrating the value of structural repetition in prediction. As shown in the table below, the ensemble of SDM, N-gram, and LPC models yielded the best overall performance.
Key insights include:
- Macro-structure is Essential: The success of the SDM approach confirms that long-term musical structure is a powerful predictor of short-term acoustic features.
- Complementary Models: Short-term (LPC) and long-term (SDM) predictors are highly complementary; their combination provides the most robust results.
- Quantization Challenges: For MFCCs, larger codebook sizes in N-gram models actually degraded performance, highlighting the difficulty of discretizing complex timbral data in real-time.
Conclusion
This work demonstrates that integrating structural repetition detection with traditional autoregressive models significantly enhances the causal prediction of continuous music features. Future research will focus on developing causal beat-tracking algorithms, improving online quantization methods, and exploring hierarchical models to better capture nested musical structures. Additionally, the authors aim to investigate the relationship between mathematical error metrics and human-perceived musical similarity.
Summary updated