Visual Narratives: Large-Scale Hierarchical Classification of Art-Historical Images

Matthias Springstein, Stefanie Schneider, Javad Rahnama, Julian Stalter, Maximilian Kristen, Eric Müller-Budack, Ralph EwerthWACV 2024

TL;DR

This paper tackles the challenge of automatically classifying complex art-historical images using the hierarchical Iconclass taxonomy. They introduce a novel Vision-Language Model (VLM) pre-training strategy using automatically generated image descriptions and a new hierarchical transformer decoder, achieving state-of-the-art results on a large-scale dataset.

What Problem Does This Paper Address?

Art historians use Iconclass, a hierarchical system, to describe the thematic content of artworks. However, manually annotating vast art collections with Iconclass is time-consuming and resource-intensive. Existing automated methods either focus on limited subsets of Iconclass or fail to fully leverage its hierarchical structure. This paper aims to bridge this gap by developing an efficient, automated image classification system that can handle the complexity and scale of the Iconclass taxonomy, making art collections more accessible for research and digital exploitation.

Methods

The authors propose a multi-stage approach:

  1. Textual Description Generation: To pre-train Vision-Language Models (VLMs), they generate textual descriptions for images using keywords from Iconclass. Three strategies are explored:
  2. Keyword (KW): Simple comma-separation of keywords.
  3. Language Models (GPT): Fine-tuning a GPT-2 model to generate more natural descriptions from keywords.
  4. Vision-Language Models (BLIP): Using the BLIP2 model with image and keywords to generate descriptions. The BLIP approach proved most effective.
  5. VLM Pre-training: They use the generated image-text pairs to pre-train a CLIP model using a contrastive loss. This step is crucial for adapting VLMs to the art domain and improving performance on rare concepts.
  6. Hierarchical Multi-label Classification (HMC): The pre-trained VLM's image encoder is used to feed into various classification heads. They propose five HMC approaches, with their novel Hierarchical Cross-Attention Transformer (CAT) being the most significant. CAT uses an encoder-decoder architecture where the decoder iteratively predicts concepts level by level, leveraging cross-attention to incorporate image features and the hierarchical structure of Iconclass. This is the first decoder designed to handle multi-label classification within a hierarchical context.

Datasets

  • ICARUS (Iconographic Classification and Representation Understanding): A newly introduced dataset comprising 477,569 images with over 1.3 million annotations for more than 20,000 unique Iconclass concepts. This dataset is significantly larger than previous ones used for Iconclass classification.
  • ICAI (Iconclass AI Test Set): An existing dataset of 87,744 images with 362,561 Iconclass concepts.

ICARUS was split into training, validation, and testing sets (80%, 10%, 10%).

Key Insights and Ideas

  • Leveraging Hierarchical Structure is Key: Simply flattening the Iconclass taxonomy (as in "Flat" classifiers) misses crucial information. Approaches that explicitly model the hierarchy (like Flat-H, Flat-W, and especially CAT) perform significantly better.
  • Domain-Specific Pre-training Matters: Pre-training VLMs on art-specific image-text pairs (generated via BLIP) yields better results than using general-purpose pre-trained models (like original CLIP on LAION-400M), particularly for less frequent concepts.
  • Automated Description Generation is Viable: The BLIP-based description generation method is effective for creating suitable training data for VLMs, overcoming the challenge of limited manual annotations.
  • Iterative Prediction for Hierarchy: The CAT model's iterative, level-by-level prediction strategy, guided by cross-attention, effectively captures the hierarchical relationships within Iconclass.

Key Results

  • The proposed CAT model achieved state-of-the-art performance on both the ICAI and ICARUS test sets, significantly outperforming baseline methods like CLIP, Flat, Flat-H, and Flat-W.
  • For example, on the ICARUS test set, the CAT model achieved a mean Average Precision (mAP) of 0.3716 for concepts with at least 1000 training images, compared to 0.2265 for the Flat classifier.
  • Pre-training with synthesized image-text pairs (using BLIP) improved performance over using the original CLIP model trained on LAION-400M.
  • Qualitative analysis with domain experts confirmed the practical utility of the CAT model, noting its ability to identify superordinate concepts even when finer-grained ones are missed, making it valuable for semi-automated annotation workflows.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their approach is promising, some images in the training set might be incompletely labeled, potentially affecting optimization. The weighting scheme in Flat-W might be less significant due to the high proportion of annotated concepts in their dataset.
  • Future Work: The authors plan to extend their approach to other hierarchical taxonomies (e.g., Chinese Iconography Thesaurus) and other hierarchical multi-label classification tasks. They also suggest exploring simultaneous optimization of VLMs and classifiers, and investigating LLM pre-training on art-historical document captions for improved description generation.

Summary updated

BibTeX
@inproceedings{springstein2024visual,
  author = {Matthias Springstein and Stefanie Schneider and Javad Rahnama and Julian Stalter and Maximilian Kristen and Eric Müller-Budack and Ralph Ewerth},
  title = {{Visual Narratives: Large-Scale Hierarchical Classification of Art-Historical Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure