Context-Aware Embeddings for Automatic Art Analysis

Noa Garcia, Benjamin Renoust, Yuta Nakashima

Abstract

Recent advancements in computer vision have enabled automated art analysis, yet most models rely solely on visual features. This paper explores the integration of contextual metadata—such as artistic school, historical period, and author influences—into neural network embeddings. By utilizing Multi-Task Learning and Knowledge Graph-based architectures, researchers achieved significant improvements in art classification and cross-modal retrieval, demonstrating that context is essential for a deeper machine understanding of art.

Introduction

Standard Convolutional Neural Networks (CNNs) typically treat paintings as isolated arrays of pixels, focusing on visual content or stylistic brushwork. However, art history is inherently contextual. A masterpiece like Picasso’s Guernica (Figure 1) cannot be fully understood through its gray cubist shapes alone; its significance is tied to the Spanish Civil War and the artist's historical influences.

Picasso's Guernica exemplifies art that requires historical context for full interpretation.

The primary challenge addressed in this research is bridging the gap between raw visual feature extraction and the interconnected metadata that defines the art world.

Methodology

The researchers proposed two distinct architectures to generate context-aware embeddings:

Multi-Task Learning (MTL) Model

The MTL approach assumes that contextual relationships are latent within visual patterns. Using a shared ResNet50 backbone, the model is trained to simultaneously identify the author, school, type, and timeframe. This hard parameter sharing forces the embedding to capture common visual elements that link these diverse artistic attributes.

Knowledge Graph Model (KGM)

The KGM approach utilizes non-visual metadata to structure the embedding space. A domain-specific Knowledge Graph (KG) was constructed, containing over 33,000 nodes (including paintings, authors, and schools) and 125,000 edges representing historical relationships (Figure 2).

A Knowledge Graph representing the interconnected relationships between artists, schools, and genres.

Using node2vec, embeddings were generated for these nodes. A visual encoder was then trained to project standard CNN features into this KG-defined space, effectively grounding visual data in historical and social context.

Experimental Setup

The models were evaluated using the SemArt dataset, which comprises 21,384 paintings. Each image is associated with metadata including Author, Title, Date, Technique, Type, School, and Timeframe. The evaluation focused on four classification tasks and two cross-modal retrieval tasks (Text-to-Art and Art-to-Text).

Results and Discussion

The inclusion of context provided a substantial performance boost across all metrics:

  • Classification Accuracy: Context-aware models consistently outperformed visual-only baselines. The KGM model was particularly effective in high-cardinality tasks, achieving 81.5% accuracy in Type classification and 61.5% in Author identification.
  • Cross-Modal Retrieval: In the "Text2Art" challenge, the KGM-Author model improved the Mean Rank (MR) by 37.24% over previous state-of-the-art methods.
  • Human-Level Performance: In challenging retrieval scenarios where all candidates belonged to the same genre (e.g., landscapes), the KGM model achieved 68% accuracy, approaching the human benchmark of 71%.

The results suggest that while MTL is better at capturing visual signatures like shared brushwork, KGM excels at modeling "arbitrary" historical relationships that may not have a clear visual manifestation.

Conclusion

This research demonstrates that context-aware embeddings significantly enhance the performance of AI in art analysis. While the current KGM approach is dependent on high-quality metadata and is domain-specific, it paves the way for hybrid architectures that combine visual relationship learning with structured knowledge. Future work will focus on integrating these two paradigms into a unified framework for more robust and flexible image analysis.

Summary updated

BibTeX
@inproceedings{garcia2019context,
  author = {Noa Garcia and Benjamin Renoust and Yuta Nakashima},
  title = {{Context-Aware Embeddings for Automatic Art Analysis}},
  booktitle = {ICMR},
  year = {2019},
  doi = {10.1145/3323873.3325028},
}
Figure