Automatic Iconographic Annotation of Medieval Illuminations: Comparing Generative Vision-Language Models and Supervised Discriminative Classifiers

Vincent Jolivet, Lucas Terriel, Chahan Vidal-Gor{è}nearXiv 2026 · arXiv

TL;DR

This paper evaluates two competing AI paradigms—generative Vision-Language Models (VLMs) and supervised discriminative classifiers (SDCs)—for the complex task of labeling medieval manuscript art. While SDCs are the "precision kings" for identifying physical objects, VLMs excel at interpreting complex scenes and generalizing across massive, hierarchical vocabularies, suggesting that the future of digital humanities lies in hybrid systems.

What Problem Does This Paper Address?

As cultural heritage institutions digitize hundreds of thousands of medieval manuscripts, the bottleneck has shifted from accessing images to indexing them. Manually describing the rich iconography of these "illuminations" is a Herculean task for human experts.

From a machine learning perspective, this is an "extreme multi-label classification" problem. The researchers used the TIMEL thesaurus, which contains over 4,000 hierarchically organized concepts. The challenge is twofold: the labels are highly imbalanced (a "long-tail" distribution where many concepts have only a few examples), and the task requires both simple perceptual recognition (e.g., "is there a dog?") and complex contextual interpretation (e.g., "is this a specific biblical scene?").

Methods

The authors compared two distinct approaches:

  1. Supervised Discriminative Classifiers (SDC): These models learn a direct mapping from images to a fixed set of labels. The researchers used a frozen DINOv3 visual backbone and tested three different "heads":
  2. Flat Linear: A simple baseline.
  3. CAT Decoder: A hierarchy-aware head that traverses the taxonomy level-by-level.
  4. ML-Decoder: A head that uses spatial attention to aggregate evidence from different parts of the image without being forced to follow a strict hierarchy.
  5. Crucially, they used Asymmetric Loss (ASL) to prevent the model from being overwhelmed by the thousands of "negative" labels for every image.
  6. Vision-Language Models (VLM): These generative models (based on Qwen3-VL-4B) "talk" about the image. The researchers fine-tuned them using LoRA (a lightweight adaptation technique) and experimented with different output formats, including flat lists and structured JSON. They also developed a "verify-calibrate" reranking pipeline that uses a "Mixture-of-Experts" approach to refine the VLM's guesses.
The VLM pipeline for TIMEL classification, illustrating the various strategies for output encoding, from flat labels to structured JSON, and the post-processing steps used to improve accuracy.

To evaluate these models fairly, the authors introduced a "soft F1" metric. Unlike standard metrics that give zero credit for a "near miss," soft F1 rewards the model if it predicts a semantically related concept (e.g., guessing "eel" instead of "salmon" is better than guessing "castle").

Datasets

  • The Image Database: 9,735 full-folio images curated by the AHLoMA research group.
  • TIMEL Thesaurus: A specialized hierarchy for medieval iconography. The study focused on a subset of 2,333 active leaf concepts.

Key Insights and Ideas

  • Loss is the Primary Lever: For SDCs, the choice of loss function (ASL) was more important for performance than the specific architecture of the classification head.
  • Hierarchy Can Be a Trap: Explicitly forcing a model to follow a taxonomic tree (like the CAT decoder) can actually limit performance. If the model makes a mistake at the top of the tree (e.g., misidentifying "animal" as "object"), it can never recover. Spatial attention (ML-Decoder) proved more robust.
  • Perception vs. Interpretation: SDCs excel at "perceptually grounded" descriptive concepts (objects, animals). VLMs, thanks to their broad pre-training on language and images, provide much better hypotheses for "interpretative" scenes that require cultural context.
  • The Alignment Challenge: The biggest hurdle for VLMs isn't "knowing" what's in the image; it's restricting their output to the specific, controlled vocabulary of the thesaurus. Structured JSON output was the most effective way to bridge this gap.

Key Results

  • Best Overall Retrieval: The ML-Decoder (SDC) was the clear winner for raw precision, achieving a precision@1 of 0.696 and a Mean Reciprocal Rank (MRR) of 0.806.
  • Best Hierarchical Coverage: The VLM (with verify-calibrate reranking) performed best on taxonomy-aware metrics, reaching a hierarchical F1 of 0.559.
  • Zero-Shot Potential: Even without specific training, strong general VLMs (like Gemini-3-Flash) captured a significant portion of the taxonomy, though they struggled with the specific indexing conventions of the dataset.

Limitations and Future Work

The authors noted that the evaluation was conducted on a relatively small subset of 46 images, which may not represent the full diversity of medieval art. Furthermore, models often over-predict "corpus co-occurrents"—for example, predicting a "shield" and "helmet" whenever a "sword" is detected, even if they aren't visible.

Future work will focus on building hybrid architectures that combine the reliable, lightweight retrieval of SDCs with the exploratory, semantic "reasoning" of VLMs to create more powerful tools for digital historians.

Summary updated

Abstract

The large-scale digitisation of medieval manuscripts creates a growing need for automated iconographic indexing. We evaluate two complementary paradigms for this task on 9,735 full-folio images annotated with 2,333 TIMEL concepts: generative Vision-Language Models (VLM) and supervised discriminative classifiers (SDC). The VLM experiments evaluate fine-tuning, structured JSON decoding, taxonomybased candidates generation, and deterministic reranking. The discriminative experiments evaluate flat, hierarchy-aware, and patch-attention heads over a frozen DINOv3 encoder. Our results highlight key functional differences: ML-Decoder (SDC) achieves the strongest overall retrieval performance (precision@1 of 0.696, MRR of 0.806), with loss calibration (Asymmetric Loss) serving as the primary performance lever. While SDCs provide superior precision and reliability within a closed vocabulary, VLMs offer strong semantic generalization and broad taxonomic coverage, particularly through structured JSON targets and verify-calibrate reranking (hierarchical F1 of 0.559). Furthermore, model behavior strongly depends on concept abstraction: SDCs excel at perceptually grounded descriptive concepts, whereas VLMs provide useful hypotheses for complex interpretative scenes. Finally, we find that enforcing explicit hierarchy within model architectures limits coverage compared to spatial feature aggregation. Overall, rather than competing approaches, SDCs and VLMs emerge as complementary components for future hybrid indexing systems combining structured retrieval with exploratory annotation assistance. As a methodological contribution, we define the soft F1 metric to provide a taxonomy-aware evaluation that credits semantically close predictions based on lowest common ancestors.
BibTeX
@misc{jolivet2026automatic,
  author = {Vincent Jolivet and Lucas Terriel and Chahan Vidal-Gor\{è\}ne},
  title = {{Automatic Iconographic Annotation of Medieval Illuminations: Comparing Generative Vision-Language Models and Supervised Discriminative Classifiers}},
  year = {2026},
  eprint = {2610.38462},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38462},
}
Figure