TL;DR
This paper introduces a novel, semiotics-inspired methodology for constructing large-scale Visual Sentiment Analysis (VSA) datasets, specifically addressing the scarcity of neutral emotion classes. Their approach leverages linguistic principles and artistic imagery to create a dataset of over 100,000 images, demonstrating superior classifier performance compared to existing datasets.
What Problem Does This Paper Address?
Visual Sentiment Analysis (VSA) aims to understand the emotions evoked by images, a field crucial for affective computing. However, building high-quality VSA datasets is challenging. Manual annotation is rigorous but slow, while automatic methods often rely on text accompanying images (like captions), which can be unreliable. A significant gap exists in datasets that include a "neutral" emotion class, forcing models to assign an emotion even when an image doesn't evoke one, or biasing them towards a specific emotion. This paper tackles the need for a scalable, accurate, and comprehensive VSA dataset that includes a neutral class, addressing the limitations of existing methods.
Methods
The core of this paper is a semiotic-based methodology for dataset construction. It leverages principles from semiotics and art theory to create textual queries that are then used to retrieve images.
- Semiotics and the Semiotic Square: The authors employ the semiotic square, a framework that analyzes meaning through oppositions. They identify pairs of opposite emotional terms (e.g., "beautiful" vs. "ugly") to define positive and negative classes.
- Neutral Class Construction: Neutrality is achieved by using queries that represent the absence of both opposing terms (e.g., "not-ugly AND not-beautiful"). To overcome search engine limitations with such queries, they use synonymous terms like "ordinary" or "normal."
- Artistic Image Integration: To enhance the dataset's ability to capture universal emotional representations, they incorporate artistic images. These are retrieved using nominalized forms of the emotional adjectives (e.g., "beauty" or "ugliness").
- Query Generation: The methodology involves pairing nouns (subjects) with these emotional adjectives to create queries. This ensures that images across different classes (positive, negative, neutral) often depict the same subject, allowing for better identification of visual patterns that evoke specific emotions.
- Curation: After retrieval, images are manually curated to ensure they match the query and are not drawings or synthetic illustrations.
Datasets
- Newly Introduced Dataset: The paper introduces a new VSA dataset containing 107,117 images, split into positive (33,176), neutral (40,716), and negative (33,176) classes.
- Existing Datasets Used for Evaluation:
- Flickr, Instagram, T4s: Large datasets with a neutral class, used for internal coherence experiments.
- IAPS, GAPED: Widely used psycho-visual experiment datasets, used for cross-dataset evaluation.
- Emoset, FI, VSO: Other large VSA datasets, used for comparison, particularly in 2-class (positive/negative) experiments.
Key Insights and Ideas
- Semiotics as a Powerful Tool for Data Generation: The paper demonstrates that semiotic principles can be systematically applied to generate effective textual queries for image retrieval, leading to high-quality, emotionally labeled datasets.
- The Importance of the Neutral Class: The inclusion of a neutral class is highlighted as crucial for more nuanced VSA, allowing models to distinguish between images that evoke emotion and those that don't.
- Artistic Imagery Enhances Generalization: Incorporating artistic images, retrieved using nominalized emotional terms, helps the dataset capture more universal visual representations of emotions, leading to better classifier generalization.
- Subject Consistency Across Classes is Key: By ensuring that images within different emotional classes often share the same primary subject, the dataset facilitates the learning of emotion-specific visual patterns rather than subject-specific ones.
Key Results
- Superior Classifier Performance: Classifiers trained on the newly constructed dataset consistently outperformed those trained on other datasets, both in terms of internal coherence (testing on the same dataset) and generalization (testing on independent datasets like IAPS and GAPED).
- CLIP Image Encoder Dominance: The CLIP Image Encoder architecture achieved the highest accuracy (up to 88.64% on their dataset) when trained on their proposed dataset.
- Matching or Surpassing Manually Labeled Datasets: The semiotic-based automatic labeling approach achieved performance comparable to or better than manually labeled datasets like FI and Emoset, even when trained on only positive and negative classes.
Limitations and Future Work
- Subjectivity and Context: The authors acknowledge that emotion perception is subjective and heavily influenced by context. While their method aims to mitigate this by focusing on visual forms and subject consistency, contextualization remains a challenge.
- Ambivalent Images: The paper notes that images evoking both positive and negative emotions (ambivalent images), which align with the "complex axis" of the semiotic square, are an important but unexplored area.
- Future Directions:
- Context-Specific Datasets: Future work could involve generating context-specific VSA datasets by incorporating contextual information into the query generation process.
- Cultural Adaptation: The methodology can be adapted to different languages, allowing for culturally diverse VSA datasets.
- Fine-Grained Labels: Extending the procedure to generate more fine-grained emotional labels is another promising avenue.
- Ambivalent Image Analysis: Further research into understanding the role of ambivalent images in visual communication is suggested.
Summary updated