EEVR: A Dataset of Paired Physiological Signals and Textual Descriptions for Joint Emotion Representation Learning

Pragya Singh, Ritvik Budhiraja, Ankush Gupta, Anshul Goswami, Mohan Kumar, Pushpendra SinghNeurIPS 2024

Introduction

Emotion recognition is often considered a "holy grail" for personalized mental health monitoring. However, current approaches face a major bottleneck: context. The EEVR dataset is a first-of-its-kind resource designed to solve this by pairing raw physiological signals (heart rate and skin conductance) with rich, self-reported textual descriptions of emotions elicited in a 360-degree Virtual Reality (VR) environment.

By applying a novel Contrastive Language-Signal Pre-training (CLSP) method, researchers demonstrate that using "language supervision" significantly improves emotion recognition accuracy and allows models to generalize to entirely different datasets in a zero-shot manner.

The Context Bottleneck

Most existing emotion datasets rely on "objective" labels—like a researcher marking a video as "stressful"—or standardized scales (e.g., a 1-to-5 rating). Unfortunately, these methods fail to capture the nuance of human experience. Two people might exhibit the exact same heart rate spike, but one feels "excited" while the other feels "anxious."

Furthermore, physiological signals like Electrodermal Activity (EDA) and Photoplethysmography (PPG) are notoriously noisy and difficult to interpret across different individuals. By ignoring the raw, descriptive text of how a person actually felt, traditional models lose the contextual "glue" that connects a physiological spike to a specific emotional state.

Bridging Biology and Language

To capture this "subjective ground truth," the researchers developed a rigorous experimental pipeline:

  1. Immersive Elicitation: 37 participants watched 360-degree VR videos designed to trigger various quadrants of the "Russell circumplex model" (varying levels of Valence/pleasure and Arousal/energy).
  2. Signal Capture: High-quality EDA (skin conductance) and PPG (blood volume pulse) were recorded using medical-grade sensors.
  3. Qualitative Interviews: After the VR sessions, participants underwent semi-structured interviews to describe their emotions in their own words. This raw text was then cleaned and paired with the corresponding signal segments.
Experimental protocol timeline showing VR exposure, physiological data collection, and qualitative interviews.

Contrastive Language-Signal Pre-training (CLSP)

Inspired by the famous CLIP model (which aligns images and text), the authors built a dual-encoder system to process this paired data:

  • Text Encoder: Uses a pre-trained DistilBERT model to process the qualitative descriptions.
  • Signal Encoder: Uses a multi-layer perceptron (MLP) to process handcrafted features from the EDA and PPG data.

The model is trained to maximize the similarity between a signal and its matching text description in a shared embedding space. Effectively, it teaches the model what "anxiety" or "joy" looks like in a heart rate wave.

Architecture of the Contrastive Language-Signal Pre-training (CLSP) model and its zero-shot classification pipeline.

Key Insights and Results

The resulting EEVR dataset features 296 emotion tasks, pairing EDA/PPG signals with textual descriptions, personality scores, and psychological well-being scores. When testing the CLSP model on EEVR and external transfer datasets (Emognition, WESAD, and NURSE), several key insights emerged:

  • Text as a "Super-Label": Raw text proved to be a more powerful supervisor than discrete labels. The model learned highly "contextualized" representations because the text captured nuances (like mixed emotions) that a simple "High Arousal" label ignores.
  • Performance Boost: Integrating text supervision via CLSP led to significant gains. For example, using EDA + Text, the model achieved a 77.25% F1-score for Arousal classification, a massive jump over signal-only baselines.
  • Zero-Shot Generalization: Remarkably, a model trained on EEVR could predict emotions in the NURSE dataset—recorded in a chaotic, real-world hospital—better than some models trained specifically on that data. On the NURSE dataset (EDA), the zero-shot CLSP model achieved 84.93% accuracy for Valence, outperforming the supervised MLP baseline (71.83%).
  • Modality Synergy: While EDA and PPG are both useful, EDA often provided a stronger signal for arousal. The combination of both, when supervised by text, yielded the most robust results.

Limitations and Future Work

While groundbreaking, the authors note several hurdles. The data was collected in a controlled lab, meaning real-world "noise" (movement, temperature changes) remains a challenge for wearable integration. Additionally, the study used a relatively small, educated, upper-middle-class demographic, which may introduce cultural biases in how emotions are described. Finally, VR headsets and finger sensors can cause physical discomfort or "VR sickness," potentially contaminating physiological data.

Looking ahead, the team suggests extending this framework to real-time monitoring via chatbots or audio-input companion apps, taking a major step toward moving emotion AI from the lab into daily life.

Summary updated

BibTeX
@inproceedings{singh2024eevr,
  author = {Pragya Singh and Ritvik Budhiraja and Ankush Gupta and Anshul Goswami and Mohan Kumar and Pushpendra Singh},
  title = {{EEVR: A Dataset of Paired Physiological Signals and Textual Descriptions for Joint Emotion Representation Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure