Exploiting CLIP-Based Multi-modal Approach for Artwork Classification and Retrieval

Alberto Baldrati, Marco Bertini, Tiberio Uricchio, Alberto Del BimboCommunications 2022 · DOI · arXiv · OpenAlex

Abstract

This research investigates the application of OpenAI’s Contrastive Language-Image Pre-training (CLIP) model to the complex tasks of artwork classification and content-based image retrieval (CBIR). By leveraging CLIP’s dual understanding of visual and textual data, the study addresses the persistent "domain shift" problem in cultural heritage—where models trained on noisy web data struggle with professional museum imagery. The authors demonstrate state-of-the-art performance on the NoisyArt dataset, highlighting CLIP's robustness and its potential for zero-shot applications in digital archives.

Introduction

The digital preservation and categorization of fine art present unique challenges for computer vision. Unlike general object recognition (e.g., identifying a "dog"), artwork recognition often requires identifying a specific instance—a unique painting or sculpture—across vastly different environments. Two primary obstacles hinder traditional models:

  1. Domain Shift: The discrepancy between "webly-supervised" training data (noisy images from Google or Flickr) and "verified" test data (clean museum photography or visitor snapshots).
  2. Instance Recognition: The need to distinguish between specific works of art rather than broad categories, often under varying lighting and angles.

The NoisyArt Dataset

To test these challenges, the researchers utilized the NoisyArt dataset, which comprises 3,120 artwork classes and approximately 89,000 images. These images are sourced from DBpedia, Google Images, and Flickr, creating a realistic scenario of data noise and stylistic variation.

Figure 1: Examples of artwork instances from the NoisyArt dataset across different web sources (DBpedia, Google, Flickr).

Methodology: Harnessing CLIP

The study explores CLIP, a model pre-trained on 400 million image-text pairs, through several experimental setups:

  • Webly-supervised Classification: Using the CLIP image encoder as a backbone for a shallow classifier fine-tuned on noisy web data.
  • Zero-shot Classification: Directly calculating similarity between images and textual descriptions without any task-specific training.
  • Multi-modal Retrieval: A pipeline where zero-shot classification identifies a text description, which is then used for text-to-image search, followed by a visual feature re-ranking phase.

Visualizing Model Attention

To interpret the model's decision-making, the authors employed GradCAM (Gradient-weighted Class Activation Mapping). This visualization technique reveals which regions of an image most influence the model's classification.

Figure 5: GradCAM visualizations showing CLIP's ability to focus on the artwork while ignoring background noise such as museum walls or bystanders.

The GradCAM results confirmed that CLIP successfully isolates the artwork from its context, effectively ignoring "background" noise like museum visitors or gallery walls when prompted with the correct description.

Key Results and Performance

The experiments yielded significant improvements over existing benchmarks:

  • Classification Accuracy: CLIP (with a ResNet-50 backbone) achieved 86.63% accuracy on the verified test set, a substantial leap from the previous state-of-the-art of 74.89%.
  • Zero-shot Capabilities: In a pure zero-shot setting, CLIP reached 60.27% accuracy, nearly doubling the performance of previous specialized models (34.93%).
  • Retrieval: The best retrieval performance (69.60 mAP) was achieved by fine-tuning CLIP features, though even the zero-shot multi-modal pipeline (47.41 mAP) outperformed standard unimodal ResNet-50 baselines (36.32 mAP).

Conclusion

This study demonstrates that multi-modal models like CLIP are exceptionally well-suited for the cultural heritage domain. Their ability to extract high-level semantics makes them robust to domain shifts and highly effective for zero-shot retrieval. While reliance on high-quality metadata remains a factor, the approach offers a powerful, scalable solution for museums and archives looking to automate the organization of their digital collections.

Summary updated

BibTeX
@inproceedings{baldrati2022exploiting,
  author = {Alberto Baldrati and Marco Bertini and Tiberio Uricchio and Alberto Del Bimbo},
  title = {{Exploiting CLIP-Based Multi-modal Approach for Artwork Classification and Retrieval}},
  booktitle = {Communications},
  year = {2022},
  doi = {10.1007/978-3-031-20302-2\_11},
}

Loading related content...

Figure