Abstract
This research investigates the challenge of automated emotional classification in abstract art, where the absence of figurative objects creates a significant "semantic gap" for traditional computer vision. By treating emotion recognition as a multimodal problem, the authors combine visual features—color and texture—with textual metadata such as titles and artist statements. The study introduces a novel joint flexible Schatten p-norm model to fuse these disparate data streams. Results across two datasets (MART and deviantArt) demonstrate that metadata serves as a critical corrective layer, significantly improving the accuracy of predicting whether a painting evokes positive or negative feelings.
Introduction: The Challenge of Abstract Emotion
In representational art, a computer can often identify the intended emotion by recognizing literal objects: a smiling face suggests happiness, while a stormy sea might evoke turbulence or fear. Abstract art, however, lacks these figurative anchors. Emotions are instead conveyed through "non-figurative" elements—the jaggedness of a line, the density of a texture, or the specific temperature of a color palette.
Psychological research indicates that humans rely heavily on titles to interpret these works; a chaotic red composition titled "Conflict" is perceived differently than one titled "Summer Sunset." This paper addresses the semantic gap in affective computing by training algorithms to utilize these same textual clues to better interpret the artist's intent, especially when visual cues are ambiguous.
Multimodal Methodology
The researchers developed a multimodal classification framework that processes two distinct streams of information:
- Visual Features: The model employs a "Bag-of-Visual-Words" approach. It extracts color features using the LAB color space and texture features via SIFT descriptors to represent the physical appearance of the artwork.
- Textual Features: Sentiment analysis is applied to titles and descriptions. For English-language metadata (from the deviantArt dataset), the researchers used SentiWordNet to calculate valence scores. For Italian metadata (from the MART dataset), words were mapped through MultiWordNet to achieve comparable sentiment metrics.
The Schatten p-norm Fusion Model
The core technical contribution of this work is the joint flexible Schatten p-norm model. In multimodal learning, the goal is to find a shared pattern between different data types (like text and images) while preserving the unique information each provides. While standard rank minimization is computationally expensive (NP-hard), the Schatten p-norm acts as a flexible proxy. It allows the model to capture the underlying structure of the fused data more efficiently than traditional late fusion methods.
Experimental Results
The study evaluated two datasets: the MART Dataset, containing 500 professional masterpieces from the Museum of Modern and Contemporary Art of Trento and Rovereto, and the deviantArt (dA) Dataset, consisting of 500 amateur works. Ground truth for emotional valence (positive vs. negative) was established through a user study using the TrueSkill ranking system.
The proposed model consistently outperformed visual-only benchmarks. On the MART dataset, the inclusion of titles improved classification accuracy from 75.3% to 78.3%. The performance boost was even more pronounced when using full artist descriptions, achieving a 3-4% improvement over standard fusion methods. This suggests that the method of data combination is as vital as the data itself.
Discussion and Key Insights
A primary finding of the research is that metadata often acts as a "correction" layer. For instance, a painting with dark colors and sharp edges—visually indicative of negative sentiment—might be titled "Spazio Inquieto" (Restless Space). The text reinforces the negative sentiment, helping the model resolve visual ambiguity. Interestingly, text was found to be more effective for the amateur deviantArt set than the professional MART set. The authors hypothesize that amateur descriptions, often written by the artists themselves to explain their feelings, provide more direct emotional signals than the historical or critical metadata associated with museum pieces.
While the study is limited by dataset size and the binary nature of the emotion scale (positive vs. negative), it paves the way for more nuanced affective computing. Future work may explore complex emotions like awe or melancholy and address the potential "lost in translation" effects when processing multilingual metadata.
Summary updated