Enhancing Human Pose Estimation in Ancient Vase Paintings via Perceptually-grounded Style Transfer Learning

Prathmesh Madhu, Angel Villar-Corrales, Ronak Kosti, Torsten Bendschus, Corinna Reinhardt, Peter Bell, Andreas K. Maier, Vincent ChristleinACM Journal on Computing and Cultural Heritage 2023 · OpenReview

Abstract

Human pose estimation (HPE) is a central part of understanding the visual narration and body movements of characters depicted in artwork collections, such as Greek vase paintings. Unfortunately, existing HPE methods do not generalise well across domains resulting in poorly recognised poses. Therefore, we propose a two-step approach: (1) adapting a dataset of natural images of known person and pose annotations to the style of Greek vase paintings by means of image style-transfer. We introduce a perceptually-grounded style transfer training to enforce perceptual consistency. Then, we fine-tune the base model with this newly created dataset. We show that using style-transfer learning significantly improves the SOTA performance on unlabelled data by more than 6% mean average precision (mAP) as well as mean average recall (mAR). (2) To improve the already strong results further, we created a small dataset (ClassArch) consisting of ancient Greek vase paintings from the 6–5th century BCE with person and pose annotations. We show that fine-tuning on this data with a style-transferred model improves the performance further. In a thorough ablation study, we give a targeted analysis of the influence of style intensities, revealing that the model learns generic domain styles. Additionally, we provide a pose-based image retrieval to demonstrate the effectiveness of our method.

The Challenge of Artistic Domains

State-of-the-art Human Pose Estimation (HPE) models are typically trained on massive datasets of natural photographs, such as COCO. However, when these models are applied to non-photorealistic domains like ancient artwork, they perform poorly. In Greek vase paintings, figures are often flat, highly stylized, and depicted with anatomical conventions that differ significantly from real-world physics.

Examples of Greek vase painting styles (black-figure and red-figure) illustrating the domain gap between natural photos and historical art.

For archaeologists and art historians, automatically detecting and analyzing these poses is crucial for understanding visual narratives—like "divine pursuit" or "wrestling" scenes—which are often defined by specific body postures. However, manually annotating enough artwork to train a deep learning model from scratch is prohibitively expensive and requires significant domain expertise.

Methodology: Bridging Photos and Vases

We propose a two-step transfer learning approach to bridge the gap between photos and vases:

  1. Synthetic Domain Adaptation: To adapt natural images to the unique aesthetic of Greek vases, we utilize Adaptive Instance Normalization (AdaIN) style transfer. We "repaint" the COCO dataset using the textures and color palettes of Greek vases, creating a synthetic dataset called Styled-COCO-Persons (SCP). Because the underlying geometry is preserved, the original ground-truth pose annotations remain valid.
  2. Perceptually-Grounded Training: During the style transfer process, we introduce a perceptual loss as a regularizer. This ensures that while the visual style changes, the human structure (the content) remains consistent, preventing the model from learning visual artifacts that could degrade pose detection.
  3. Fine-tuning: We then fine-tune these "styled" models on a small, high-quality dataset of real vase paintings using a Faster R-CNN for person detection and HRNet-W32 for keypoint localization.

The ClassArch Dataset

To evaluate our approach, we introduced ClassArch, a new benchmark consisting of 1,513 images from 6th–5th century BCE Greek vases. These images were manually annotated with person bounding boxes and pose keypoints, serving as the "gold standard" for our experiments. We found that training on styled data makes the model a "faster learner"; a model pre-trained on SCP and then fine-tuned on only 50% of the ClassArch dataset performed as well as a standard model trained on 100% of the data.

Performance and Results

Our final "Styled-Tuned" model reached 67.2 mAP on the ClassArch dataset, outperforming models tuned directly from standard photo-based weights (65.6 mAP). Qualitatively, the improvement is stark, as the model correctly identifies keypoints that baseline methods like OpenPose miss entirely due to the stylized rendering of the figures.

Comparison of pose estimation results on a Greek vase: (a) Original image, (b) OpenPose baseline showing failure to localize keypoints, and (c) our proposed method demonstrating accurate estimation.

Beyond detection, we demonstrated the effectiveness of our method through a pose-based image retrieval system. By measuring the similarity between detected postures, we successfully retrieved similar narrative scenes (e.g., finding all "fleeing" characters) across a large collection of unlabelled vases, providing a powerful tool for digital humanities research.

Limitations and Future Work

While pose estimation has seen significant gains, person detection remains challenging when figures are heavily intertwined, such as in Greek wrestling (Agonal) scenes. Future work will explore incorporating geometric priors or semantic segmentation to better separate characters from decorative backgrounds and improve detection in complex compositions.

Summary updated

Abstract

Human pose estimation (HPE) is a central part of understanding the visual narration and body movements of characters depicted in artwork collections, such as Greek vase paintings. Unfortunately, existing HPE methods do not generalise well across domains resulting in poorly recognised poses. Therefore, we propose a two step approach: (1) adapting a dataset of natural images of known person and pose annotations to the style of Greek vase paintings by means of image style-transfer. We introduce a perceptually-grounded style transfer training to enforce perceptual consistency. Then, we fine-tune the base model with this newly created dataset. We show that using style-transfer learning significantly improves the SOTA performance on unlabelled data by more than 6% mean average precision (mAP) as well as mean average recall (mAR). (2) To improve the already strong results further, we created a small dataset (ClassArch) consisting of ancient Greek vase paintings from the 6–5th century BCE with person and pose annotations. We show that fine-tuning on this data with a style-transferred model improves the performance further. In a thorough ablation study, we give a targeted analysis of the influence of style intensities, revealing that the model learns generic domain styles. Additionally, we provide a pose-based image retrieval to demonstrate the effectiveness of our method. The code and pretrained models can be found at https://github.com/angelvillar96/STLPose.
BibTeX
@article{madhu2023enhancing,
  author = {Prathmesh Madhu and Angel Villar-Corrales and Ronak Kosti and Torsten Bendschus and Corinna Reinhardt and Peter Bell and Andreas K. Maier and Vincent Christlein},
  title = {{Enhancing Human Pose Estimation in Ancient Vase Paintings via Perceptually-grounded Style Transfer Learning}},
  journal = {ACM Journal on Computing and Cultural Heritage},
  year = {2023},
  url = {https://openreview.net/forum?id=u1SO1O3V8-},
}
Figure