TL;DR
This study investigates how different music genres affect brain activity by analyzing hemodynamic responses using fNIRS. The researchers developed a deep learning model that can accurately classify music genres and predict emotional responses, suggesting fNIRS is a promising tool for understanding music's impact on the brain and for personalized music recommendations.
What Problem Does This Paper Address?
Understanding how music influences our emotions and cognitive states is a long-standing area of interest, with implications for music therapy, affective computing, and even personalized entertainment. While self-reports and facial expressions are common ways to gauge emotional responses, they can be subjective and easily manipulated. Physiological signals, however, offer a more objective and involuntary measure of our internal states. This paper specifically addresses the challenge of using functional near-infrared spectroscopy (fNIRS) to capture and interpret the subtle hemodynamic changes in the brain that occur when individuals listen to different types of music. The goal is to determine if these brain signals can reliably distinguish between music genres and reflect emotional responses, thereby paving the way for more sophisticated brain-computer interfaces and emotion-aware systems.
Methods
The study employed a combination of machine learning and deep learning techniques to analyze fNIRS data.
- fNIRS Data Acquisition: Cerebral hemodynamic responses were measured using the Obelab NIRSIT device, which records changes in oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) concentrations in the prefrontal cortex.
- Data Preprocessing: Raw fNIRS signals underwent filtering, noise channel rejection, and normalization using the Modified Beer-Lambert law to obtain HbO2 and HbR data.
- Feature Extraction (for traditional ML): For K-Nearest Neighbors (KNN) and Random Forest (RF) models, statistical features were extracted from both the time and frequency domains of the pre-processed signals.
- Deep Learning Model: A 1D Convolutional Neural Network (CNN) was utilized for automatic feature extraction directly from the time-series fNIRS data.
- Ensemble Learning: To address inter-participant variability and potential issues with uneven data quality, a stacked ensemble model was developed. This involved training individual sub-models for each participant and then using a meta-learner to combine their predictions.
- Classification Tasks: Two main classification tasks were performed:
- Classifying music genre (classical, instrumental, pop).
- Classifying subjective emotional responses based on participant ratings (converted into three-class categories: negative, neutral, positive).
- Evaluation Metrics: Classification performance was evaluated using accuracy, precision, recall, and F1-score, with weighted averages used for imbalanced datasets.
Datasets
The study utilized a newly collected dataset comprising fNIRS signals from 27 participants listening to 12 different music pieces across three genres (classical, instrumental, pop). This dataset is not explicitly made publicly available in the paper, but the methodology describes its collection and characteristics.
Key Insights and Ideas
- Deep Learning Outperforms Traditional ML: The 1D CNN model, which automatically extracts features, significantly outperformed traditional machine learning methods (KNN, RF) in classifying both music genres and subjective emotional responses. This highlights the power of deep learning for complex physiological signal analysis.
- HbO2 and HbR Synergy: Combining both oxyhemoglobin (HbO2) and deoxyhemoglobin (HbR) signals yielded better classification accuracy than using either signal alone, suggesting a more comprehensive understanding of brain activity.
- fNIRS as a Reliable Indicator of Emotion: The study demonstrated that fNIRS signals can reliably reflect participants' emotional responses to music, aligning with subjective ratings and even revealing unexpected emotional reactions to certain stimuli.
- Lingering Music Effects: The analysis of offset lengths suggested that hemodynamic responses to music can persist even after the music has finished, potentially due to participants recalling their emotional experience.
- Challenging Music Assumptions: The research revealed that common assumptions about music's effects (e.g., binaural beats for focus) might not always align with actual physiological and verbal responses, emphasizing the need for objective physiological data.
Key Results
- Music Genre Classification: The 1D CNN model achieved 73.4% accuracy in classifying the three music genres (classical, instrumental, pop) using both HbO2 and HbR signals.
- Subjective Emotion Classification: The highest accuracy for classifying subjective emotional responses was 80.5% for the "irritating → soothing" emotion scale using the 1D CNN model.
- Traditional ML Performance: KNN and RF models achieved lower accuracies, with the best ternary classification accuracy at 59.4% and binary classification at 74.9%.
- Visualizing Brain Activity: Timeline analysis of HbO2 signals showed that responses became more prominent after a few minutes of listening and exhibited distinct patterns for different genres, particularly in the mid and right pre-frontal cortex.
- Qualitative Findings: Participants' verbal comments often aligned with their physiological responses, even when these differed from expected emotional reactions to certain music pieces.
Limitations and Future Work
- Small Sample Size: The authors acknowledge that the number of participants (27) might be considered small for training deep learning models and plan to collect more data in the future.
- Feature Engineering Challenges: While deep learning reduced the need for manual feature engineering, the authors noted that traditional methods yielded poor results, suggesting further investigation into feature selection techniques.
- Device Sensitivity: The fNIRS device's sensitivity to movement and noise led to the elimination of many channels, impacting data quality. Future work will explore advanced preprocessing techniques like wavelet and Fourier transformations.
- Ensemble Model Limitations: The stacked ensemble model assumes all participants contribute useful information, which may not always be true if some participants' models have poor accuracy. Future work will involve grid search and optimization for ensemble model selection.
- Music Stimuli Selection: The selection of instrumental and pop music stimuli was based on the researchers' judgment due to a lack of expert annotations, potentially introducing bias. Future experiments aim to gather expert annotations for music stimuli.
- Future Directions: The authors suggest exploring a wider range of music stimuli, comparing musical instruments with digital instruments, investigating the lingering effects of music, and developing advanced wearable devices for continuous monitoring and personalized music recommendations.
Summary updated