ArcAid: Analysis of Archaeological Artifacts using Drawings

Offry Hayon, Stefan Münger, Ilan Shimshoni, Ayellet Tal

Abstract

Archaeology is an intriguing domain for computer vision. It suffers not only from shortage in (labeled) data, but also from highly-challenging data, which is often extremely abraded and damaged. This paper proposes a novel semi-supervised model for classification and retrieval of images of archaeological artifacts. This model utilizes unique data that exists in the domain -- manual drawings made by special artists. These are used during training to implicitly transfer the domain knowledge from the drawings to their corresponding images, improving their classification results. We show that while learning how to classify, our model also learns how to generate drawings of the artifacts, an important documentation task, which is currently performed manually. Last but not least, we collected a new dataset of stamp-seals of the Southern Levant. Our code and dataset are publicly available.

Introduction: The Challenge of Damaged Artifacts

Archaeological artifacts present a unique set of challenges for standard computer vision models. Unlike the clear, standardized objects found in datasets like ImageNet, ancient artifacts—such as the stamp-seals of the Southern Levant—are frequently eroded, broken, or stained. Furthermore, because these items were handcrafted across different centuries and regions, there is significant intra-class variation; for instance, two "Lion" seals may look entirely different depending on their origin.

Examples of raw input images of ancient stamp-seals showing significant abrasion and surface noise.

Currently, archaeologists rely on manual drawings created by specialists. These drawings are "interpretive," meaning the artist uses their expertise to highlight intended shapes while filtering out cracks or surface noise. ArcAid proposes a way to use these expert drawings to "teach" a machine to see what the archaeologist sees.

Methodology: Multi-Modal Learning with Privileged Information

The ArcAid framework utilizes a semi-supervised multi-modal model that treats drawings as a source of "privileged information" during training. The architecture employs dual encoders: one for raw images ($Enc_{Im}$) and one for expert drawings ($Enc_{Draw}$).

During training, the model minimizes a similarity loss (negative cosine similarity) between the embeddings of an image and its corresponding drawing. This forces the image encoder to ignore "noise" (like cracks) and focus on the structural features highlighted in the drawing. The model is trained on three simultaneous tasks:

  1. Classification: Predicting the shape (e.g., beetle, bird) or historical period using cross-entropy loss.
  2. Similarity: Aligning image and drawing representations in latent space.
  3. Generation: A decoder ($Dec_{Im}$) attempts to reconstruct the drawing from the image embedding, ensuring the embedding preserves enough spatial detail to be useful for documentation.

Crucially, at test time, the model only requires the raw image. The drawings serve as "scaffolding" during the training phase to build a more robust image encoder.

Results and Discussion

The model was evaluated on the new Corpus of the Stamp-seals of the Southern Levant (CSSL) dataset, which contains 6,636 pairs of images and drawings.

The ArcAid framework demonstrates successful classification by shape and period, alongside the generation of high-quality archaeological drawings from raw input.

ArcAid significantly outperformed standard backbones, improving shape classification accuracy by 7.1% to 18.3%. On the CSSL dataset, it achieved 90.8% accuracy for shape classification and 84.5% for historical period classification. Beyond classification, the model's ability to generate drawings provides a valuable tool for documentation, producing results that are more archaeologically accurate than standard edge detectors (like DexiNed), as it successfully completes faint edges while ignoring surface textures.

Saliency maps (Grad-CAM) revealed that models trained only on images often focus on the "torso" of an animal. Models trained with ArcAid shifted their focus to diagnostic features like horns or head shapes, mimicking how human experts classify items.

Conclusion

ArcAid demonstrates that expert domain knowledge, encapsulated in manual drawings, can be effectively transferred to computer vision models. By aligning noisy image data with clean, interpretive drawings, the system achieves state-of-the-art performance in artifact classification and retrieval while automating the labor-intensive task of archaeological drawing generation.

Summary updated

Abstract

Archaeology is an intriguing domain for computer vision. It suffers not only from shortage in (labeled) data, but also from highly-challenging data, which is often extremely abraded and damaged. This paper proposes a novel semi-supervised model for classification and retrieval of images of archaeological artifacts. This model utilizes unique data that exists in the domain -- manual drawings made by special artists. These are used during training to implicitly transfer the domain knowledge from the drawings to their corresponding images, improving their classification results. We show that while learning how to classify, our model also learns how to generate drawings of the artifacts, an important documentation task, which is currently performed manually. Last but not least, we collected a new dataset of stamp-seals of the Southern Levant. Our code and dataset are publicly available.
BibTeX
@inproceedings{hayon2024arcaid,
  author = {Offry Hayon and Stefan Münger and Ilan Shimshoni and Ayellet Tal},
  title = {{ArcAid: Analysis of Archaeological Artifacts using Drawings}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=O7ddiXwsY1},
}
Figure