ArtFace: Towards Historical Portrait Face Identification via Model Adaptation

Francois Poh, Anjith George, Sébastien MarcelarXiv 2025 · DOI · arXiv

Introduction

Identifying the subjects of historical portraits, known as "sitters," is a fundamental task for art historians that has traditionally relied on subjective visual analysis. Automating this process presents a significant challenge for computer vision due to the "domain shift" between modern photography and historical art. Standard facial recognition (FR) systems, trained on massive datasets of digital photographs, struggle with the stylistic nuances of paintings, such as brushwork, lighting, and the aging of materials. These artistic factors lead to high intra-class variation, where the same individual appears markedly different across various works, and low inter-class variation, where different individuals painted in the same era or style appear deceptively similar.

Methodology

To address these challenges, the ArtFace framework proposes a multi-pronged adaptation strategy that bridges the gap between photography and artistic media through a dual-stream architecture.

1. CLIP-LoRA Adaptation

The system leverages the CLIP (Contrastive Language-Image Pre-training) foundation model. Unlike standard FR models, CLIP is trained on a vast, diverse corpus of internet data, allowing it to capture broad stylistic concepts. To adapt CLIP to the specific domain of historical portraits without requiring a massive labeled dataset, the researchers employed Low-Rank Adaptation (LoRA). This technique inserts small, trainable rank-decomposition matrices into the model's attention layers while keeping the original weights frozen, enabling efficient fine-tuning on limited data.

2. FR Model Adaptation

In parallel, the system utilizes a state-of-the-art FR model (IResNet100). To preserve its structural knowledge of facial geometry while adapting to the artistic domain, the authors performed minimal adaptation by fine-tuning only the final fully connected (FC) layer.

The ArtFace architecture combines CLIP-LoRA and FR model adaptation via feature fusion.

3. Training and Fusion

Both streams are optimized using a Triplet Loss function with hard negative mining. This approach forces the model to minimize the distance between an "anchor" image and a "positive" image (the same person) while maximizing the distance to a "negative" image (a different person). Hard negative mining specifically targets the most confusing pairs—such as different people painted in the same style—to refine the model's discriminative power. The final identification is performed by concatenating the normalized embeddings from both the adapted CLIP and FR models, creating a fused representation that captures both geometric structure and stylistic context.

Experimental Results

The ArtFace framework was evaluated on the Historical Faces dataset, comprising 766 paintings of 210 sitters. The results demonstrate that the fusion of adapted models significantly outperforms baseline systems and commercial off-the-shelf (COTS) solutions.

ROC curves illustrating the performance gains of the ArtFace fusion model compared to baselines.

Key quantitative findings include:

  • Error Reduction: The base IResNet100 model's Equal Error Rate (EER) of 14.0% was reduced to 9.9% in the final fused model.
  • Strict Accuracy: At a strict False Accept Rate (FAR) of 0.1%, the True Accept Rate (TAR) jumped from 29.9% (base) to 39.7% (fusion).
  • Ablation Insights: Fine-tuning CLIP with LoRA alone improved its individual EER from 17.9% to 13.1%, confirming the effectiveness of parameter-efficient adaptation for artistic domains.

Conclusion

ArtFace demonstrates that foundation models like CLIP can serve as effective "style interpreters" when combined with traditional facial recognition architectures. By fusing geometric precision with stylistic awareness, the model achieves a synergistic boost in performance. Future work will explore the use of synthetic data generation to overcome the scarcity of labeled historical portraits and more advanced domain adaptation techniques to handle heterogeneous media, such as comparing sketches to oil paintings.

Summary updated

Abstract

Identifying sitters in historical paintings is a key task for art historians, offering insight into their lives and how they chose to be seen. However, the process is often subjective and limited by the lack of data and stylistic variations. Automated facial recognition is capable of handling challenging conditions and can assist, but while traditional facial recognition models perform well on photographs, they struggle with paintings due to domain shift and high intra-class variation. Artistic factors such as style, skill, intent, and influence from other works further complicate recognition. In this work, we investigate the potential of foundation models to improve facial recognition in artworks. By fine-tuning foundation models and integrating their embeddings with those from conventional facial recognition networks, we demonstrate notable improvements over current state-of-the-art methods. Our results show that foundation models can bridge the gap where traditional methods are ineffective. Paper page at https://www.idiap.ch/paper/artface/
BibTeX
@misc{poh2025artface,
  author = {Francois Poh and Anjith George and Sébastien Marcel},
  title = {{ArtFace: Towards Historical Portrait Face Identification via Model Adaptation}},
  year = {2025},
  eprint = {2508.20626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.20626},
}
Figure