Emotion detection from handwriting and drawing samples using an attention-based transformer model

Zohaib Ahmad Khan, Yuanqing Xia, Khursheed Aurangzeb, Fiza Khaliq, Mahmood Alam, Javed Ali Khan, Muhammad Shahid AnwarPeerJ Comput. Sci 2024 · DOI

TL;DR

This paper introduces an attention-based transformer model for emotion detection using handwriting and drawing samples. By integrating these two modalities and leveraging the power of self-attention, the model achieves state-of-the-art results, demonstrating a promising new avenue for understanding human emotions beyond traditional text or speech analysis.

What Problem Does This Paper Address?

Accurately understanding and interpreting human emotions is a complex but crucial task with applications spanning mental health, user experience, and more. While various methods exist, many focus on single modalities like facial expressions, voice, or text. This paper tackles the challenge of emotion detection (ED) by exploring the underutilized modalities of handwriting and drawing. These actions are deeply intertwined with motor and cognitive processes, and thus can offer unique insights into an individual's emotional state. However, existing research has often focused on single modalities, and the complexity and overlap of emotional states make precise detection difficult, especially when relying solely on text. This work aims to bridge this gap by developing a model that can effectively analyze both handwriting and drawing to infer emotions.

Methods

The core of this research is an attention-based transformer model. This architecture is well-suited for sequential data and excels at capturing long-range dependencies, which are crucial for understanding the nuances in handwriting and drawing strokes.

Here's a breakdown of the key methodological aspects:

  • Multimodal Input: The model uniquely integrates features extracted from both handwriting and drawing samples. This multimodal approach allows for a more comprehensive analysis than single-modality methods.
  • Feature Extraction: Time-domain, frequency-domain, and statistical features are extracted from the handwriting and drawing signals. This includes metrics like mean, standard deviation, spectral entropy, and autocorrelation.
  • Transformer Architecture: The model utilizes a transformer architecture, known for its self-attention mechanisms. This allows the model to weigh the importance of different parts of the input sequence, focusing on the most informative features for emotion prediction.
  • Self-Attention: The self-attention layer computes a weighted sum of embeddings, where the weights are determined by the similarity between different parts of the input. This helps the model identify intricate patterns and dependencies.
  • Multi-Head Attention: To capture diverse aspects of the input, the self-attention mechanism is extended to include multiple heads, allowing the model to attend to different feature subspaces simultaneously.
  • Classification: The final output of the transformer is passed through a linear layer and a softmax function to generate a probability distribution over the predicted emotion categories.

Datasets

The study utilizes two datasets for training and evaluation:

  • EMOTHAW Dataset: This dataset contains samples from 129 individuals (aged 21-32) whose emotional states (anxiety, depression, stress) were measured using the DASS assessment. It includes seven types of activities recorded on a digitizing tablet: drawing pentagons and houses, handwriting words, drawing circles and clocks, and copying a phrase. The data includes pen azimuth, altitude, pressure, time stamp, and position.
  • SemEval Dataset: This dataset comprises 4,359 handwritten text samples with emotional annotations, sourced from news headlines from reputable sources. The emotions are categorized according to Ekman's six fundamental emotions: happiness, sadness, fear, surprise, anger, and disgust.

Key Insights and Ideas

  • Synergy of Modalities: The paper strongly suggests that combining handwriting and drawing data significantly enhances emotion detection accuracy compared to using either modality alone. This highlights the complementary nature of these behavioral cues.
  • Transformer's Advantage: The attention-based transformer model proves highly effective at capturing long-range dependencies and subtle patterns in the data, outperforming traditional machine learning techniques. This is particularly beneficial for the sequential and spatial nature of handwriting and drawing.
  • Beyond Basic Emotions: While many ED systems focus on a few basic emotions, this work demonstrates the capability to identify a broader range of emotional states, including more nuanced ones like anxiety and stress, from behavioral data.
  • Potential for Mental Health: The high accuracy achieved, especially in identifying depression and anxiety, points to a significant potential for this technology in mental health diagnostics and personalized well-being assessments.

Key Results

The proposed attention-based transformer model achieved impressive results on both datasets:

  • EMOTHAW Dataset: The model achieved a peak accuracy of 92.64% for depression detection when integrating both handwriting and drawing features. For anxiety, it reached 83.22% (both modalities), and for stress, 79.41% (writing features). These results surpassed conventional machine learning techniques.
  • SemEval Dataset: For text-based emotion detection, the model achieved state-of-the-art results, with a notable F1 score of 87.06% for sad state identification. It also achieved high F1 scores for angry (83.12%) and happy (79.73%) states.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Quality and Bias: The model's performance is contingent on the quality and representativeness of the training data, which can introduce biases and affect generalizability.
  • Limited Diversity in Styles: The training data might not fully capture the vast diversity in individual handwriting and drawing styles, potentially impacting adaptability to extreme variations.
  • Cultural Nuances: Emotional expression can vary across cultures, and the model's performance might differ in diverse cultural contexts.
  • Machine Translation: Reliance on machine translation for languages beyond the training set can introduce errors.
  • Predefined Emotions: The model is trained on a specific set of emotions and may not capture the full spectrum of human emotional expression.
  • Ethical Considerations: Privacy and consent are critical concerns when deploying emotion detection technologies.

For future work, the authors suggest exploring transfer learning techniques. Pre-training the attention-based transformer model on large datasets and then fine-tuning it for specific emotion detection tasks could potentially improve performance, especially on smaller datasets.

Summary updated

BibTeX
@article{khan2024emotion,
  author = {Zohaib Ahmad Khan and Yuanqing Xia and Khursheed Aurangzeb and Fiza Khaliq and Mahmood Alam and Javed Ali Khan and Muhammad Shahid Anwar},
  title = {{Emotion detection from handwriting and drawing samples using an attention-based transformer model}},
  journal = {PeerJ Comput. Sci.},
  year = {2024},
  doi = {10.7717/peerj-cs.1887},
}
Figure