TL;DR
This paper evaluates two competing AI paradigms—generative Vision-Language Models (VLMs) and supervised discriminative classifiers (SDCs)—for the complex task of labeling medieval manuscript art. While SDCs are the "precision kings" for identifying physical objects, VLMs excel at interpreting complex scenes and generalizing across massive, hierarchical vocabularies, suggesting that the future of digital humanities lies in hybrid systems.
What Problem Does This Paper Address?
As cultural heritage institutions digitize hundreds of thousands of medieval manuscripts, the bottleneck has shifted from accessing images to indexing them. Manually describing the rich iconography of these "illuminations" is a Herculean task for human experts.
From a machine learning perspective, this is an "extreme multi-label classification" problem. The researchers used the TIMEL thesaurus, which contains over 4,000 hierarchically organized concepts. The challenge is twofold: the labels are highly imbalanced (a "long-tail" distribution where many concepts have only a few examples), and the task requires both simple perceptual recognition (e.g., "is there a dog?") and complex contextual interpretation (e.g., "is this a specific biblical scene?").
Methods
The authors compared two distinct approaches:
- Supervised Discriminative Classifiers (SDC): These models learn a direct mapping from images to a fixed set of labels. The researchers used a frozen DINOv3 visual backbone and tested three different "heads":
- Flat Linear: A simple baseline.
- CAT Decoder: A hierarchy-aware head that traverses the taxonomy level-by-level.
- ML-Decoder: A head that uses spatial attention to aggregate evidence from different parts of the image without being forced to follow a strict hierarchy.
- Crucially, they used Asymmetric Loss (ASL) to prevent the model from being overwhelmed by the thousands of "negative" labels for every image.
- Vision-Language Models (VLM): These generative models (based on Qwen3-VL-4B) "talk" about the image. The researchers fine-tuned them using LoRA (a lightweight adaptation technique) and experimented with different output formats, including flat lists and structured JSON. They also developed a "verify-calibrate" reranking pipeline that uses a "Mixture-of-Experts" approach to refine the VLM's guesses.
To evaluate these models fairly, the authors introduced a "soft F1" metric. Unlike standard metrics that give zero credit for a "near miss," soft F1 rewards the model if it predicts a semantically related concept (e.g., guessing "eel" instead of "salmon" is better than guessing "castle").
Datasets
- The Image Database: 9,735 full-folio images curated by the AHLoMA research group.
- TIMEL Thesaurus: A specialized hierarchy for medieval iconography. The study focused on a subset of 2,333 active leaf concepts.
Key Insights and Ideas
- Loss is the Primary Lever: For SDCs, the choice of loss function (ASL) was more important for performance than the specific architecture of the classification head.
- Hierarchy Can Be a Trap: Explicitly forcing a model to follow a taxonomic tree (like the CAT decoder) can actually limit performance. If the model makes a mistake at the top of the tree (e.g., misidentifying "animal" as "object"), it can never recover. Spatial attention (ML-Decoder) proved more robust.
- Perception vs. Interpretation: SDCs excel at "perceptually grounded" descriptive concepts (objects, animals). VLMs, thanks to their broad pre-training on language and images, provide much better hypotheses for "interpretative" scenes that require cultural context.
- The Alignment Challenge: The biggest hurdle for VLMs isn't "knowing" what's in the image; it's restricting their output to the specific, controlled vocabulary of the thesaurus. Structured JSON output was the most effective way to bridge this gap.
Key Results
- Best Overall Retrieval: The ML-Decoder (SDC) was the clear winner for raw precision, achieving a precision@1 of 0.696 and a Mean Reciprocal Rank (MRR) of 0.806.
- Best Hierarchical Coverage: The VLM (with verify-calibrate reranking) performed best on taxonomy-aware metrics, reaching a hierarchical F1 of 0.559.
- Zero-Shot Potential: Even without specific training, strong general VLMs (like Gemini-3-Flash) captured a significant portion of the taxonomy, though they struggled with the specific indexing conventions of the dataset.
Limitations and Future Work
The authors noted that the evaluation was conducted on a relatively small subset of 46 images, which may not represent the full diversity of medieval art. Furthermore, models often over-predict "corpus co-occurrents"—for example, predicting a "shield" and "helmet" whenever a "sword" is detected, even if they aren't visible.
Future work will focus on building hybrid architectures that combine the reliable, lightweight retrieval of SDCs with the exploratory, semantic "reasoning" of VLMs to create more powerful tools for digital historians.
Summary updated