This paper, presented at EVA London 2021, addresses a long-standing friction between the computer vision (CV) and art history communities. While computer scientists have spent years building tools for style classification and authentication, many art historians remain unimpressed, viewing these tools as solutions to 19th-century problems. Author Amalia Foka argues that for CV to be truly useful, it must move beyond "looking" at art and start "reading" it the way modern historians do—through the lens of critical theory and historical context.
TL;DR
The paper identifies a disconnect between computer vision research and modern art history, proposing a shift from simple style classification to "visual word searching" of complex historical and social features. By analyzing top art history journals, the author maps specific scholarly needs—like identifying clothing materials or political landmarks—to existing CV tasks, providing a roadmap for future interdisciplinary research.
What Problem Does This Paper Address?
The core issue is a "gap in understanding." Computer scientists often treat art history as a series of well-defined visual problems: Who painted this? What style is it? Is it a fake? These tasks fall under "Connoisseurship" and "Formalism," methods that dominated the field a century ago.
However, since the 1980s, art history has been transformed by critical theory (semiotics, Marxism, feminism, etc.). Modern historians are less interested in the "pixels" and more interested in the "historicity"—the social, political, and religious conditions under which an artwork was created. They don't want an AI to interpret the art for them; they want a sophisticated retrieval system that allows them to find, chain, and browse images based on complex semantic features that trigger new research questions.
Methods
Instead of proposing a new algorithm, the author introduces a methodology for interdisciplinary translation.
- Literature Review: The author systematically studied articles from the three highest-ranked art history journals (The Art Bulletin, Art Journal, and Art History) published in 2017 and 2018.
- Feature Extraction: By analyzing how historians actually use images in these papers, the author identified specific visual "features" that drive modern research.
- CV Mapping: These features were then translated into technical CV problems (e.g., material recognition, OCR, geo-localization) to show that current technology is already capable of meeting these needs if redirected.
Datasets
The "dataset" for this study was the corpus of academic art history literature from 2017–2018. While the paper does not introduce a new image dataset, it references several existing CV benchmarks that could be repurposed for art history, including:
- COCO-Text: For text detection and recognition.
- DeepFashion: For robust clothing recognition and retrieval.
- ClickBAIT-v2: For real-time object detection.
Key Insights and Ideas
- The "Visual Word Search": Art historians want a visual equivalent of "Ctrl+F." They need to search for specific objects or themes across massive databases to see how they evolve over time.
- Signatures are Metadata: Beyond authentication, the language of a signature (e.g., Chinese characters vs. Latin alphabet) or its placement on a canvas is used by historians to study identity, sovereignty, and the artist's presence.
- Materiality as History: Historians look at the type of fabric in a painting (cotton vs. linen) to infer trade relations, class status, and even links to the history of slavery.
- Visual Culture Intertextuality: Research often involves finding "visual culture" images (like brand trademarks or protest signs) embedded within fine art to track the transmission of social movements.
- Temporality through Architecture: Recognizing architectural styles from different epochs within a single painting allows historians to study memory, forgetting, and ethnic identity.
Key Results
The paper provides a taxonomy of features that art historians actually care about, mapped to CV tasks:
- Text/Signatures: Can be addressed via OCR and script identification to study authorship and national identity.
- Clothing/Materials: Can be addressed via semantic attribute recognition to infer class and global trade contexts.
- Landmarks/Plants: Can be addressed via geo-localization and computational botany to study colonialism and cross-cultural exchange.
- Facial Recognition: Can be extended beyond "royals and artists" to identify recurring mythological and biblical figures.
The author concludes that CV research is "adequately advanced" to handle these tasks; the missing link is a unified system that integrates these features into a searchable interface for researchers.
Limitations and Future Work
- Copyright Hurdles: The "well-known issue" of copyright for artwork images remains a significant barrier to building the large-scale databases required for these systems.
- The "Black Box" Problem: Art historians are wary of automated judgments. Future systems must allow users to contribute to the definition and training of concepts, enabling them to "fine-tune" the AI according to specific research questions.
- Integration: The next major step is the materialization of a unified image search and retrieval system that combines all these disparate CV capabilities into one tool.
Summary updated