TL;DR
This paper introduces a new task: emotional stimuli detection in artworks, aiming to pinpoint the specific regions within an artwork that evoke emotions. To facilitate this, they've created APOLO, the first pixel-level annotated dataset for this task, and evaluated several baseline models, revealing the significant challenges and limitations of current approaches.
What Problem Does This Paper Address?
Understanding human emotions is a complex and subjective endeavor. While previous research has focused on recognizing emotions evoked by artworks or generating emotional captions, this paper tackles a more granular problem: identifying where in an artwork the emotional impact originates. Artworks, unlike photographs, present unique challenges due to diverse artistic styles and the subjective nature of emotional responses. This means a single artwork can evoke different emotions in different viewers, or even different emotions from different parts of the same artwork. Existing datasets and methods haven't adequately addressed this "emotional stimuli detection" at a pixel level, leaving a gap in our understanding of how visual elements translate into emotional experiences.
Methods
The core of this work is the introduction of a new task and dataset. The authors propose emotional stimuli detection as a task requiring models to output a segmentation map indicating regions that evoke a specific emotion. They frame this as a segmentation problem, where the model learns to predict a segment $s$ for a given artwork $a$ and emotion $e$, i.e., $s = f_e(a)$. They also consider a variant with reference, where an utterance $u$ provides additional context: $s = f_e(a, u)$.
To tackle this, they developed a novel annotation process for their dataset, APOLO. This process involves:
- Phrase-Region Selection: Annotators identify noun phrases in utterances that correspond to visual concepts in the artwork and mark their locations.
- Region Annotation: Annotators draw pixel-level masks for these identified regions.
- Aggregation: Annotations are aggregated to form a final emotional stimuli map.
They also evaluated several baseline models, including:
- With-Reference Baselines: Models like VilBERT and 12-in-1 fine-tuned on visual grounding tasks (refCOCO), and CLIP + VinVL, which leverage utterance-region similarity.
- Without-Reference Baselines: Object detection models (FasterRCNN, VinVL) and saliency detection models (CASNet, CASNet II).
- Weakly-Supervised Model (WESD): A novel model designed specifically for this task, trained using utterances from the ArtEmis dataset as pseudo ground truth.
Datasets
The primary dataset introduced in this paper is APOLO (Artwork Provoked emotion EvaLuation).
- Origin: APOLO is built upon the existing ArtEmis dataset, utilizing its test set.
- Content: It contains 6781 emotional stimuli annotations across 4718 artworks, covering eight emotions (amusement, anger, awe, contentment, disgust, excitement, fear, sadness).
- Key Feature: APOLO is the first dataset to provide pixel-level annotations of emotional stimuli in artworks, offering a more precise understanding of emotion elicitation compared to previous datasets.
- Existing Datasets Mentioned: ArtEmis [11] (for emotional captions and utterances), EmotionROI [30] and EMOd [31] (for pixel-level emotional stimuli, but smaller and focused on social media images).
Key Insights and Ideas
- Subjectivity is Key: The inherent subjectivity of art appreciation means that a single artwork can evoke multiple emotions, and the "emotional stimuli" can vary significantly between viewers. This makes the task inherently challenging and distinct from standard image analysis.
- Utterances as Proxies: While utterances (textual explanations of emotions) are helpful, they don't always translate directly to typical vision-language tasks. The authors found that using emotion labels as input to vision-language models sometimes performed worse than using no text reference at all, suggesting a domain gap.
- Emotional Stimuli Coincide with Objects/Attributes: The success of object detection baselines (FasterRCNN, VinVL) indicates that emotional stimuli often align with specific objects or attributes within an artwork, which is consistent with how utterances describe these elements.
- CASNet's Center Bias: The CASNet model, trained on saliency data, tends to focus on central regions of images, which might not always align with the true emotional stimuli in artworks.
- WESD's Promise: The proposed weakly-supervised model, WESD, shows promising results, outperforming some baselines and demonstrating an ability to capture relevant emotional regions without direct pixel-level supervision during training.
- Generative Models Struggle: Even advanced generative models like Stable Diffusion have difficulty precisely linking emotional prompts to specific regions in generated artworks, highlighting the need for better understanding of emotional stimuli.
Key Results
- Dataset Size: APOLO contains 6781 emotional stimuli annotations across 4718 artworks.
- Baseline Performance: Quantitative results (Table 3) show that none of the baseline models achieve perfect performance.
- For the with-reference task, CLIP + VinVL with utterances achieved the best segmentation score (Pr@50: 40.08), but WESD (without reference) performed better (Pr@50: 44.97).
- For the without-reference task, WESD achieved the highest segmentation score (Pr@50: 44.97), followed closely by VinVL (Pr@50: 43.67).
- Qualitative Findings:
- VinVL tends to generate many small bounding boxes around objects, while WESD produces larger regions that often encompass more of the relevant emotional area.
- CASNet often predicts regions near the center of the image, potentially missing stimuli at the edges.
- WESD's emotion-wise predictions show that certain regions can evoke multiple emotions, and emotions like "awe" and "contentment" tend to involve larger, more scenic regions.
- Stable Diffusion struggles to focus attention on specific emotional stimuli when prompted, often attending to corners or less relevant areas.
Limitations and Future Work
- Subjectivity and Domain Gap: The authors acknowledge the inherent subjectivity of emotions and the potential domain gap between typical vision-language tasks and art analysis.
- Ethical Concerns: They highlight the ethical considerations of emotion prediction, including privacy issues and potential model abuse, and commit to responsible data handling and mitigation strategies.
- Model Performance: Despite the new dataset and baselines, current models do not achieve satisfactory performance, indicating the task's difficulty and the need for further research.
- Future Work:
- Investigating how generative models like Stable Diffusion can be improved by guiding them to focus on emotional stimuli.
- Exploring the connection between natural images and paintings to understand cross-domain learning for emotion.
- Developing more robust models that can better handle the nuances of subjective emotional responses in artworks.
- Further research into the ethical implications and responsible use of emotion analysis technologies.
Summary updated