Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style

Marvin Limpijankit, Milad Alshomary, Yassin Oulad Daoud, Amith Ananthram, Tim Trombley, Elias Stengel-Eskin, Mohit Bansal, Noam M. Elcott, Kathleen McKeown

Abstract

Vision-Language Models (VLMs) have become increasingly proficient at a range of computer vision tasks, including analyzing and generating art. In an interdisciplinary collaboration between computer scientists and art historians, we characterize the mechanisms underlying VLMs' ability to predict artistic style and assess the extent to which they align with the criteria art historians use to reason about artistic style. We employ a latent-space decomposition approach to identify concepts that drive art style prediction and conduct quantitative evaluations, causal analysis, and assessment by art historians. Our findings indicate that 73% of the extracted concepts are judged by art historians to exhibit a coherent and semantically meaningful visual feature, and 90% of concepts used to predict the style of a given artwork were judged relevant. In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success; for example, the model might "understand" a concept in more formal terms, such as dark/light contrasts.

Introduction

Vision-Language Models (VLMs) like Qwen and LLaVA have demonstrated remarkable proficiency in identifying artistic styles. However, their decision-making process often remains a "black box." Unlike simple object recognition, artistic style is an abstract quality grounded in a complex interplay of texture, color, and composition. This research addresses a central question: Are these models learning to "see" like human experts, or are they relying on statistical shortcuts unrelated to art history? Understanding this alignment is crucial for ensuring that AI tools used in the humanities reason in ways that are consistent with domain expertise.

Methodology

To bridge the gap between computer science and art history, the researchers developed a multi-stage interpretability pipeline:

  1. Patch-Level Concept Decomposition: Artworks were split into 4x4 grids. Using Semi-Nonnegative Matrix Factorization (Semi-NMF), the researchers decomposed the model's internal activations into discrete "concepts." This allowed for the identification of specific visual features—such as the soft rendering of drapery or architectural elements—that the model prioritizes.
  2. Linear Probing: A simple linear classifier was trained to determine if these extracted concepts could accurately predict the model's final style classification.
  3. Causal Intervention: To move beyond correlation, the team performed "surgical" interventions by subtracting specific concept vectors from the model's hidden state to measure the resulting drop in style confidence (e.g., Baroque), proving a causal link.
  4. Expert Evaluation: Six art historians labeled and rated the coherence and relevance of the AI’s internal concepts.
Figure 5: A grid of visual concept patches extracted from artworks, illustrating the variety of features—from drapery to figures—identified by the decomposition framework.

Key Insights and Results

The "Formalist" AI

A fascinating finding emerged when AI and humans disagreed: the AI often employed "formal" logic. For instance, a model might correctly identify a painting as "Realism" using a concept that art historians initially found irrelevant. Upon closer inspection, historians realized the model was detecting specific dark/light contrasts (chiaroscuro) that exist in the image but are not the primary human criteria for that specific patch.

Detail Bias and Thematic Shortcuts

The model frequently associates specific content with style. For example, "forest details" were heavily linked to "Romanticism," even when appearing in "Realist" paintings. This suggests the model occasionally relies on thematic shortcuts rather than pure stylistic form.

Figure 6: An example of a specific architectural concept patch identifying Gothic tracery, a feature used by the model to distinguish architectural styles.

Quantitative Performance

  • High Interpretability: Art historians judged 73% of the AI-extracted concepts to be semantically meaningful.
  • Strong Alignment: 90% of the concepts used to predict a style were deemed "relevant" by human experts.
  • Predictive Power: Linear probes using these concepts achieved up to 95% accuracy in predicting the model’s style output, suggesting the decomposition faithfully captured the model's internal logic.
  • Model Comparison: Qwen3 emerged as a top performer, significantly outperforming LLaVA-1.5, which showed a strong bias toward predicting "Baroque" or "Romanticism."

Conclusion

While VLMs show strong alignment with expert reasoning, they also exhibit a "non-human" logic based on formal visual properties. The study highlights a domain shift challenge: concepts learned at the local patch level do not always aggregate perfectly to the global image level. Future work will focus on scaling this approach to more diverse artistic traditions and developing methods to aggregate local insights into a holistic understanding of entire compositions, mimicking the way an art historian moves their gaze from a single brushstroke to the whole canvas.

Summary updated

Abstract

VLMs have become increasingly proficient at a range of computer vision tasks, such as visual question answering and object detection. This includes increasingly strong capabilities in the domain of art, from analyzing artwork to generation of art. In an interdisciplinary collaboration between computer scientists and art historians, we characterize the mechanisms underlying VLMs' ability to predict artistic style and assess the extent to which they align with the criteria art historians use to reason about artistic style. We employ a latent-space decomposition approach to identify concepts that drive art style prediction and conduct quantitative evaluations, causal analysis and assessment by art historians. Our findings indicate that 73% of the extracted concepts are judged by art historians to exhibit a coherent and semantically meaningful visual feature and 90% of concepts used to predict style of a given artwork were judged relevant. In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success; for example, the model might "understand" a concept in more formal terms, such as dark/light contrasts.
BibTeX
@misc{limpijankit2026does,
  author = {Marvin Limpijankit and Milad Alshomary and Yassin Oulad Daoud and Amith Ananthram and Tim Trombley and Elias Stengel-Eskin and Mohit Bansal and Noam M. Elcott and Kathleen McKeown},
  title = {{Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style}},
  year = {2026},
  eprint = {2603.11024},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.11024},
}
Figure