Abstract
Archaeology is an intriguing domain for computer vision. It suffers not only from shortage in (labeled) data, but also from highly-challenging data, which is often extremely abraded and damaged. This paper proposes a novel semi-supervised model for classification and retrieval of images of archaeological artifacts. This model utilizes unique data that exists in the domain -- manual drawings made by special artists. These are used during training to implicitly transfer the domain knowledge from the drawings to their corresponding images, improving their classification results. We show that while learning how to classify, our model also learns how to generate drawings of the artifacts, an important documentation task, which is currently performed manually. Last but not least, we collected a new dataset of stamp-seals of the Southern Levant. Our code and dataset are publicly available.
Introduction: The Challenge of Damaged Artifacts
Archaeological artifacts present a unique set of challenges for standard computer vision models. Unlike the clear, standardized objects found in datasets like ImageNet, ancient artifacts—such as the stamp-seals of the Southern Levant—are frequently eroded, broken, or stained. Furthermore, because these items were handcrafted across different centuries and regions, there is significant intra-class variation; for instance, two "Lion" seals may look entirely different depending on their origin.
Currently, archaeologists rely on manual drawings created by specialists. These drawings are "interpretive," meaning the artist uses their expertise to highlight intended shapes while filtering out cracks or surface noise. ArcAid proposes a way to use these expert drawings to "teach" a machine to see what the archaeologist sees.
Methodology: Multi-Modal Learning with Privileged Information
The ArcAid framework utilizes a semi-supervised multi-modal model that treats drawings as a source of "privileged information" during training. The architecture employs dual encoders: one for raw images ($Enc_{Im}$) and one for expert drawings ($Enc_{Draw}$).
During training, the model minimizes a similarity loss (negative cosine similarity) between the embeddings of an image and its corresponding drawing. This forces the image encoder to ignore "noise" (like cracks) and focus on the structural features highlighted in the drawing. The model is trained on three simultaneous tasks:
- Classification: Predicting the shape (e.g., beetle, bird) or historical period using cross-entropy loss.
- Similarity: Aligning image and drawing representations in latent space.
- Generation: A decoder ($Dec_{Im}$) attempts to reconstruct the drawing from the image embedding, ensuring the embedding preserves enough spatial detail to be useful for documentation.
Crucially, at test time, the model only requires the raw image. The drawings serve as "scaffolding" during the training phase to build a more robust image encoder.
Results and Discussion
The model was evaluated on the new Corpus of the Stamp-seals of the Southern Levant (CSSL) dataset, which contains 6,636 pairs of images and drawings.
ArcAid significantly outperformed standard backbones, improving shape classification accuracy by 7.1% to 18.3%. On the CSSL dataset, it achieved 90.8% accuracy for shape classification and 84.5% for historical period classification. Beyond classification, the model's ability to generate drawings provides a valuable tool for documentation, producing results that are more archaeologically accurate than standard edge detectors (like DexiNed), as it successfully completes faint edges while ignoring surface textures.
Saliency maps (Grad-CAM) revealed that models trained only on images often focus on the "torso" of an animal. Models trained with ArcAid shifted their focus to diagnostic features like horns or head shapes, mimicking how human experts classify items.
Conclusion
ArcAid demonstrates that expert domain knowledge, encapsulated in manual drawings, can be effectively transferred to computer vision models. By aligning noisy image data with clean, interpretive drawings, the system achieves state-of-the-art performance in artifact classification and retrieval while automating the labor-intensive task of archaeological drawing generation.
Summary updated