Abstract
Traditional methods for affective video annotation often struggle with the "affective gap" between content features and human emotional response. This paper introduces "brainsourcing," a framework that aggregates implicit EEG signals from a crowd of viewers to automatically generate affective metadata. By leveraging the "wisdom of the crowd," the proposed method significantly improves the accuracy and confidence of emotional state predictions (valence and arousal). Experimental results on the DEAP and SEED datasets demonstrate that while individual brain responses are noisy, collective signals can achieve high-fidelity annotations, reaching up to 80% accuracy with a crowd of 14 participants.
Introduction
To enhance recommendation systems and multimedia retrieval, content must be tagged with affective metadata reflecting human emotional responses. Current approaches rely on manual annotation, which is resource-intensive, or content-based AI, which often fails to capture subjective emotional nuances. Brainsourcing offers a middle ground: using EEG data from multiple participants to create a reliable, automated consensus label. This study investigates whether aggregating brain signals can bypass explicit surveys and provide a robust measure of a video's emotional impact.
Methodology
The brainsourcing framework involves recording EEG data while participants watch video stimuli. The technical pipeline includes:
- Feature Extraction: 18 features (e.g., Power Sum, Entropy) were extracted across theta, alpha, beta, and gamma frequency bands for each EEG channel.
- Classification: A linear Support-Vector Machine (SVM) served as the base classifier to predict binary dimensions of affect: Valence (positive vs. negative) and Arousal (high vs. low intensity).
- Experimental Setups: Three models were tested: Leaving One Video Out (LOVO) for content generalization, Leaving One Subject Out (LOSO) for user generalization, and Subject-Dependent Models (SDM) for personalized performance.
- Fusion Strategy: Crowd predictions were derived via majority voting over averaged predicted probabilities. A "crowdsourcing confidence" metric was introduced using a softmax function to quantify collective certainty.
The study utilized two benchmarks: DEAP (32 participants, 40 music videos, subjective labels) and SEED (15 participants, 10 film clips, curated stimuli).
Results and Discussion
The primary finding is that affective decoding accuracy scales significantly with crowd size. This "wisdom of the crowd" effect allows the system to filter out individual neural noise, leaving a clear signal of the content's dominant emotional impact.
As shown in the figure, both accuracy and confidence increase as more participants are added to the crowd. In the DEAP dataset, individual valence accuracy of approximately 58% improved to ~70% with 6 participants and reached ~80% with 14 participants. The SEED dataset, featuring more curated stimuli, showed even faster convergence, with accuracy saturating near 100% with small crowd sizes.
The analysis also revealed that "crowdsourcing-effective" videos—those eliciting strong, consistent emotional responses—are more easily annotated. Approximately 65–72% of videos in the DEAP dataset met this criteria. Furthermore, the researchers identified a "label leakage" risk and proposed a more rigorous evaluation model that separates the labeling group from the crowdsourcing group to ensure realistic performance metrics.
Conclusion and Future Work
Brainsourcing demonstrates a viable path for automated affective annotation by aggregating implicit neural responses. While current results are based on controlled lab settings and binary labels, the scaling laws observed suggest that larger crowds can overcome the inherent noise in individual EEG signals. Future work will focus on continuous emotional scales, consumer-grade wearables, and unsupervised methods to further generalize these findings.
Summary updated