Abstract
As virtual environments continue to advance, the demand for immersive and emotionally engaging experiences has grown. Addressing this demand, we introduce Emotion enabled Virtual avatar mapping using Optimized KnowledgE distillation (EVOKE), a lightweight emotion recognition framework designed for the seamless integration of emotion recognition into 3D avatars within virtual environments. Our approach leverages knowledge distillation involving multi-label classification on the publicly available DEAP dataset, which covers valence, arousal, and dominance as primary emotional classes. Remarkably, our distilled model, a CNN with only two convolutional layers and 18 times fewer parameters than the teacher model, achieves competitive results, boasting an accuracy of 87% while demanding far less computational resources. This equilibrium between performance and deployability positions our framework as an ideal choice for virtual environment systems. Furthermore, the multi-label classification outcomes are utilized to map emotions onto custom-designed 3D avatars.
Introduction
The evolution of virtual reality (VR) and digital twins has significantly enhanced user immersion, yet these systems often lack "emotional intelligence"—the ability to perceive and respond to a user's affective state. While Electroencephalogram (EEG) signals provide a rich source of data for emotion detection, the deep learning models required for high-accuracy recognition are typically computationally intensive. This creates a significant bottleneck for real-time applications, where massive neural networks can cause latency in resource-heavy VR environments. To address this, we present EVOKE, a framework that bridges the gap between high-fidelity emotion recognition and real-time deployment on edge devices.
Methodology
The EVOKE framework utilizes Optimized Knowledge Distillation to compress a complex teacher model into a highly efficient student model.
Feature Engineering and Representation
Raw EEG signals from the DEAP dataset are preprocessed into Differential Entropy (DE) features across four frequency bands: alpha, beta, gamma, and theta. To preserve the spatial relationships of the electrodes on the scalp, these features are organized into a 3D grid representation. This spatial awareness is crucial, as convolutional architectures are better suited than global attention mechanisms (like Transformers) for capturing local spatial nuances in EEG data.
Teacher-Student Framework
The distillation process involves two models:
- Teacher Model: A Continuous Convolutional Neural Network (CCNN) with four layers, providing high accuracy at the cost of computational complexity.
- Student Model: A lightweight CNN consisting of only two convolutional layers.
The student is trained to mimic the "soft" probability outputs of the teacher using a temperature-adjusted sigmoid activation function. This approach allows the student to learn the nuanced relationships and correlations between emotional dimensions rather than just hard labels.
Multi-Label Classification
Unlike traditional models that predict a single discrete emotion, EVOKE performs multi-label classification across three dimensions: Valence (pleasantness), Arousal (intensity), and Dominance (degree of control). This VAD model provides a more granular representation of human emotion.
Results and Discussion
The EVOKE framework demonstrates a remarkable balance between efficiency and performance.
Efficiency and Speed
The distilled student model achieves an 18x reduction in parameters compared to the teacher (353K vs. 6.2M). This reduction translates directly to inference speed; the student model processes samples in approximately 0.33 ms, making it significantly faster than the teacher and well-suited for real-time VR integration. The high throughput—over 80,000 samples per second—ensures that emotion recognition does not become a system bottleneck.
Accuracy and Mapping
Despite its reduced size, the student model maintains a competitive accuracy of 87.62% on the DEAP dataset. The predicted VAD values are then mapped to eight discrete emotional states to drive the expressions of custom 3D avatars. The inclusion of the Dominance dimension is particularly valuable, as it allows the system to distinguish between emotions that appear similar in 2D space, such as "Angry" versus "Frustrated."
Conclusion
EVOKE provides a robust and efficient solution for integrating emotion recognition into virtual environments. By leveraging optimized knowledge distillation, the framework achieves high-fidelity emotion mapping from EEG signals while maintaining the low computational footprint required for real-time deployment. Future work will focus on transitioning from discrete emotion mapping to continuous, fluid facial animations and testing the system with real-world, noisy EEG data from wearable headsets.
Summary updated