Abstract
As an emerging sub-field of music information retrieval (MIR), music imagery information retrieval (MIIR) aims to retrieve information from brain activity recorded during music cognition—such as listening to or imagining music pieces. This is a highly inter-disciplinary endeavor that requires expertise in MIR as well as cognitive neuroscience and psychology. The OpenMIIR initiative strives to foster collaborations between these fields to advance the state of the art in MIIR. As a first step, electroencephalography (EEG) recordings of music perception and imagination have been made publicly available, enabling MIR researchers to easily test and adapt their existing approaches for music analysis like fingerprinting, beat tracking or tempo estimation on this new kind of data. This paper reports on first results of MIIR experiments using these OpenMIIR datasets and points out how these findings could drive new research in cognitive neuroscience.
Introduction
While retrieval of information from audio signals is a mature field, Music Imagery Information Retrieval (MIIR)—extracting information from brain activity—remains a significant challenge. The primary obstacles include a low signal-to-noise ratio in EEG data, which is often contaminated by muscle movements and background cognitive processes, and the lack of interdisciplinary collaboration between MIR experts and neuroscientists. The OpenMIIR initiative addresses these issues by providing open-access datasets and benchmarks for standard MIR tasks, such as song identification and tempo estimation, performed directly on brain waves.
Methods
The research evaluates three computational strategies for decoding musical information from EEG:
- ERP-Inspired Tempo Analysis: This method utilizes autocorrelation on the EEG signal to identify periodicities corresponding to the musical beat, even within single trials.
- Linear Envelope Reconstruction: This approach attempts to reconstruct the audio's amplitude envelope from EEG using linear decoders. However, the complexity and lack of sparsity in musical signals make this more difficult than in speech decoding.
- Similarity-Constraint Encoding (SCE): A novel triplet-based representation learning technique. A convolutional neural network is trained to map EEG fragments of the same song to similar feature vectors while distancing fragments of different songs. This pre-training helps the model focus on stimulus-related activity rather than noise.
Dataset
The OpenMIIR Dataset includes EEG recordings from 10 participants. The stimuli consisted of 12 short music fragments (7–16 seconds) across three categories: songs with lyrics, instrumental versions of those songs, and purely instrumental pieces. Data was collected under three conditions: stimulus perception (listening), cued imagination (with a metronome), and uncued imagination.
Results and Discussion
The SCE-based model demonstrated significant improvements in stimulus identification, achieving 27.59% accuracy (chance is 8.33%). It also successfully distinguished between 3/4 and 4/4 meters with 69.44% accuracy. Tempo estimation was feasible with errors between 7 and 12 BPM, notably performing better for songs with lyrics.
Analysis of the neural network weights revealed that internal activations naturally peaked at musical downbeats, suggesting the model learned rhythmic structures without explicit supervision. Furthermore, confusion matrices showed the model often confused songs with their instrumental counterparts, indicating it was responding to musical content (melody/tempo) rather than non-musical artifacts.
Conclusion
While results for music perception are promising, decoding musical imagination remains difficult due to temporal variance—the "imagination gap" where subjects imagine at varying speeds. Future research will involve larger datasets and more complex architectures, such as Recurrent Neural Networks (RNNs), to better account for the dynamic nature of musical imagination.
Summary updated