EEVR: A Dataset of Paired Physiological Signals and Textual Descriptions for Joint Emotion Representation Learning

EEVR (Emotion Elicitation in Virtual Reality) is a novel dataset specifically designed for language supervision-based pre-training of emotion recognition tasks, such as valence and arousal classification. It features high-quality physiological signals, including electrodermal activity (EDA) and photoplethysmography (PPG), acquired through emotion elicitation via 360-degree virtual reality (VR) videos. Additionally, it includes subject-wise textual descriptions of emotions experienced during each stimulus gathered from qualitative interviews. Emotional stimuli are selected to induce diverse emotions across all four quadrants of Russell's circumplex model.

The dataset consists of recordings from 37 participants and is the first dataset to pair raw text with physiological signals, providing additional contextual information that objective labels cannot offer. To leverage this dataset, we introduced the Contrastive Language Signal Pre-training (CLSP) method, which jointly learns representations using pairs of physiological signals and textual descriptions.

Our results show that integrating self-reported textual descriptions with physiological signals significantly improves performance on emotion recognition tasks, such as arousal and valence classification. Moreover, our pre-trained CLSP model demonstrates strong zero-shot transferability to existing datasets, outperforming supervised base line models, suggesting that the representations learned by our method are more contextualized and generalized. The dataset also includes baseline models for arousal, valence, and emotion classification, as well as code for data cleaning and feature extraction.

Kindly fill the following form to get access to the full dataset: form link.
Codes for EEVR can be found here.

The Architecture for Contrastive-Language Singal Pre-Training (CLSP)

To underscore the importance of integrating textual descriptions in emotion recognition, we introduce the Contrastive Language-Signal Pre-training (CLSP) method for extracting more contextualized representations.

The model was trained on physiological signals and text pairs to learn a joint embedding space, where both modalities are closely aligned using a contrastive loss function. Following pre-training, we evaluated the model's performance on test subject data using the leave-one-subject-out cross-validation approach.
CLSP uses separate neural networks to process two types of input data: physiological signals (PPG and EDA) and text. For signal data, it uses linear, hidden layers sizes of 50 and 100, while for text, it applies a pre-trained DistilBERT model. The model then optimizes a contrastive objective to increase similarity in positive pairs and decrease it in negative ones. We observed that CLSP significantly improved emotion recognition in arousal and valence tasks compared to models trained only on signal data. This underscores the value of adding text-based supervision to enhance emotion representation from physiological signals.

Results for Physiological Baseline without text using Hand-crafted features +NN and with text using CLSP on 296 text-signal pairs (seed=43, epoch=15)

EEVR: A Virtual Reality-Based Emotion Dataset Featuring Paired Physiological Signals and Textual Descriptions © 2024 by Pragya Singh, Pushpendra Singh is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-sa/4.0/.