TL;DR
This paper introduces the Visualized Emotion Ontology (VEO), a novel machine-readable model that semantically defines 25 emotions and links them to visual representations using shapes, lines, and colors. This work aims to improve human-computer interaction, particularly in healthcare, by enabling intelligent agents to better understand and respond to user emotions.
What Problem Does This Paper Address?
Healthcare interactions are often highly emotional, and physicians' ability to understand and respond to patient emotions is crucial for well-being. While emotions are fundamental to human decision-making and actions, representing them in a way that machines can understand and utilize is challenging. Existing models often focus on physiological cues or complex cognitive processes, making them difficult for computational systems to process. This paper addresses the need for a structured, machine-readable representation of emotions that is grounded in visual cues, making it more accessible for applications like conversational agents in clinical settings.
Methods
The core of this research is the development of the Visualized Emotion Ontology (VEO), built upon the revised Ortony, Clore, & Collins (OCC) model of emotions. The VEO is structured as a polyarchy with five branches: Action, Aspect, Consequence, Emotion, and Visualization. The "Emotion" branch defines 25 emotions, and the "Visualization" branch links these emotions to visual elements. This involves:
- Semantic Definition: Emotions are defined using OWL (Web Ontology Language), a standard for representing knowledge in a machine-readable format. This leverages the OCC model's approach of defining emotions based on situations and affective reactions.
- Visual Representation Design: Drawing from existing research on the psychological associations of colors, shapes, and lines with emotions, the authors designed unique visual motifs for each of the 25 emotions. For instance, positive emotions are generally represented by circles with curved lines, while negative emotions use downward-pointing triangles with sharp lines. Colors are also strategically chosen to align with common emotional associations (e.g., yellow for happiness, blue for sadness).
- Ontology Evaluation: The VEO's quality was assessed using the Burton-Jones' semiotic metric suite, measuring syntactic, semantic, and pragmatic aspects.
- Crowdsourced Validation: A survey was conducted to validate the designed visualizations, asking participants to rate how well the visual representations matched the intended emotions.
Datasets
- Existing Literature: The VEO's design and visualizations are informed by existing research on emotion models (Ekman, OCC), and the psychological associations of colors, shapes, and lines with emotions.
- Crowdsourced Survey Data: A survey was administered to 1082 participants recruited via Amazon Mechanical Turk (MTurk). This dataset was used to validate the accuracy of the emotion visualizations. Participants rated the match between an emotion and its corresponding visual representation.
Key Insights and Ideas
- Bridging the Gap: The VEO successfully bridges the gap between abstract emotional concepts and concrete visual representations, making emotions more interpretable for both humans and machines.
- Visual Language of Emotion: The research demonstrates that specific combinations of colors, shapes, and lines can reliably evoke or represent distinct emotions, providing a foundation for a visual language of emotion.
- Healthcare Applications: The VEO's structured nature and visual components offer a promising avenue for enhancing patient-provider communication in healthcare, potentially through embodied conversational agents that can express empathy visually.
- Beyond Basic Emotions: By incorporating 25 emotions, the VEO moves beyond the limited set of "basic" emotions often studied, offering a richer representation of the emotional spectrum.
Key Results
- Ontology Quality: The VEO demonstrated strong performance in ontology evaluation metrics, achieving above-average machine-readability, linguistic quality, and domain coverage compared to a sample of cognitive ontologies. The overall quality score was 0.85.
- Visualization Validation: The crowdsourced survey indicated that for 17 out of 25 emotions, participants agreed that the visual representation matched the intended emotion. For 16 of these, the results were statistically significant (p < 0.002).
- Specific Emotion Visualizations: For example, "joy" is visualized as a yellow circle with double curved orange lines, and "distress" as a blue downward-pointing triangle with sharp lines. "Surprise" is represented as a yellow upward-pointing triangle with double sharp blue lines.
Limitations and Future Work
- Limitations:
- For eight emotions, more participants disagreed than agreed with the visualization. The paper suggests this might be due to the specific incorrect emotion-image pairs used in the survey or the chosen emotion names.
- The current model does not consider the use of motion, which could further enhance the visualizations.
- While the VEO aligns with the OCC model, it includes some emotions not present in the EMO ontology and vice versa.
- Future Work:
- Expand the VEO by creating subclasses for nuanced emotion types (e.g., different levels of fear).
- Incorporate affective terms from existing resources like WN-AFFECT.
- Add instances to represent individual user emotions.
- Further investigate the eight emotions that did not fully support the visualizations by comparing them with different incorrect emotion-image pairs.
- Develop software that interfaces with the VEO and employs it in conversational agents, potentially processing user emotions from various modalities (utterances, expressions).
- Explore the use of motion in visualizations.
Summary updated