Looking at Mondrian's Victory Boogie-Woogie: What Do I Feel?

Andreza Sartori, Yan Yan, Gözde Özbal, Alkim Almila Akdag Salah, Albert Ali Salah, Nicu Sebe

Abstract

This research investigates the challenge of automated emotional classification in abstract art, where the absence of figurative objects creates a significant "semantic gap" for traditional computer vision. By treating emotion recognition as a multimodal problem, the authors combine visual features—color and texture—with textual metadata such as titles and artist statements. The study introduces a novel joint flexible Schatten p-norm model to fuse these disparate data streams. Results across two datasets (MART and deviantArt) demonstrate that metadata serves as a critical corrective layer, significantly improving the accuracy of predicting whether a painting evokes positive or negative feelings.

Introduction: The Challenge of Abstract Emotion

In representational art, a computer can often identify the intended emotion by recognizing literal objects: a smiling face suggests happiness, while a stormy sea might evoke turbulence or fear. Abstract art, however, lacks these figurative anchors. Emotions are instead conveyed through "non-figurative" elements—the jaggedness of a line, the density of a texture, or the specific temperature of a color palette.

Figure 1: An example of abstract art where emotions are conveyed through non-figurative elements like sharp lines and geometric shapes.

Psychological research indicates that humans rely heavily on titles to interpret these works; a chaotic red composition titled "Conflict" is perceived differently than one titled "Summer Sunset." This paper addresses the semantic gap in affective computing by training algorithms to utilize these same textual clues to better interpret the artist's intent, especially when visual cues are ambiguous.

Multimodal Methodology

The researchers developed a multimodal classification framework that processes two distinct streams of information:

  1. Visual Features: The model employs a "Bag-of-Visual-Words" approach. It extracts color features using the LAB color space and texture features via SIFT descriptors to represent the physical appearance of the artwork.
  2. Textual Features: Sentiment analysis is applied to titles and descriptions. For English-language metadata (from the deviantArt dataset), the researchers used SentiWordNet to calculate valence scores. For Italian metadata (from the MART dataset), words were mapped through MultiWordNet to achieve comparable sentiment metrics.

The Schatten p-norm Fusion Model

The core technical contribution of this work is the joint flexible Schatten p-norm model. In multimodal learning, the goal is to find a shared pattern between different data types (like text and images) while preserving the unique information each provides. While standard rank minimization is computationally expensive (NP-hard), the Schatten p-norm acts as a flexible proxy. It allows the model to capture the underlying structure of the fused data more efficiently than traditional late fusion methods.

Experimental Results

The study evaluated two datasets: the MART Dataset, containing 500 professional masterpieces from the Museum of Modern and Contemporary Art of Trento and Rovereto, and the deviantArt (dA) Dataset, consisting of 500 amateur works. Ground truth for emotional valence (positive vs. negative) was established through a user study using the TrueSkill ranking system.

Figure 4: Accuracy (Acc) of the proposed model across different values of the p-norm and the regularization parameter lambda_2.

The proposed model consistently outperformed visual-only benchmarks. On the MART dataset, the inclusion of titles improved classification accuracy from 75.3% to 78.3%. The performance boost was even more pronounced when using full artist descriptions, achieving a 3-4% improvement over standard fusion methods. This suggests that the method of data combination is as vital as the data itself.

Discussion and Key Insights

A primary finding of the research is that metadata often acts as a "correction" layer. For instance, a painting with dark colors and sharp edges—visually indicative of negative sentiment—might be titled "Spazio Inquieto" (Restless Space). The text reinforces the negative sentiment, helping the model resolve visual ambiguity. Interestingly, text was found to be more effective for the amateur deviantArt set than the professional MART set. The authors hypothesize that amateur descriptions, often written by the artists themselves to explain their feelings, provide more direct emotional signals than the historical or critical metadata associated with museum pieces.

While the study is limited by dataset size and the binary nature of the emotion scale (positive vs. negative), it paves the way for more nuanced affective computing. Future work may explore complex emotions like awe or melancholy and address the potential "lost in translation" effects when processing multilingual metadata.

Summary updated

Abstract

Abstract artists use non-figurative elements (i.e. colours, lines, shapes, and textures) to convey emotions and often rely on the titles of their various compositions to generate (or enhance) an emotional reaction in the audience. Several psychological works observed that the metadata (i.e., titles, description and/or artist statements) associated with paintings increase the understanding and the aesthetic appreciation of artworks. In this paper we explore if the same metadata could facilitate the computational analysis of artworks, and reveal what kind of emotional responses they awake. To this end, we employ computer vision and sentiment analysis to learn statistical patterns associated with positive and negative emotions on abstract paintings. We propose a multimodal approach which combines both visual and metadata features in order to improve the machine performance. In particular, we propose a novel joint flexible Schatten p- norm model which can exploit the sharing patterns between visual and textual information for abstract painting emotion analysis. Moreover, we conduct a qualitative analysis on the cases in which metadata help improving the machine performance.
BibTeX
@inproceedings{sartori2015looking,
  author = {Andreza Sartori and Yan Yan and Gözde Özbal and Alkim Almila Akdag Salah and Albert Ali Salah and Nicu Sebe},
  title = {{Looking at Mondrian's Victory Boogie-Woogie: What Do I Feel?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure