Beyond the Pipeline: Analyzing Key Factors in End-to-End Deep Learning for Historical Writer Identification

Hanif Rasyidi, Moshiur Farazi

TL;DR

This paper dissects the factors influencing end-to-end deep learning for historical writer identification (HWI), a challenging task due to document degradation and limited data. They found that while end-to-end models struggle with generalization, a specific combination of pre-processing (binarization + Text-AOI selection), a SwinV2-Base backbone, and ArcFace loss achieves state-of-the-art performance with a simpler pipeline.

What Problem Does This Paper Address?

Historical Writer Identification (HWI) aims to determine the author of historical documents. This is significantly harder than modern writer identification due to factors like document aging, ink fading, damage, and the inherent variability in handwriting over time and across individuals. Traditional methods rely on manual feature engineering and are brittle. While end-to-end deep learning promises automation, existing models often fail to generalize to real-world scenarios, especially when test writers are not seen during training (zero-shot learning). This paper investigates why these end-to-end systems falter and identifies key components for robust HWI.

Methods

The authors frame HWI as a pipeline with three core stages:

  • Pre-processing: They explore three strategies:
  • SIFT-based keypoint detection: A traditional method for extracting local features.
  • Historical Document Binarization: Using a deep learning model to create a text mask, separating text from background noise.
  • Text-AOI Selection: A novel approach that combines binarization with connected component analysis and dilation to precisely isolate relevant text regions, discarding noise and non-textual elements.
  • Feature Extraction (Model Selection & Loss Function):
    • Backbones: They compare Transformer models (SwinV2 variants) and CNNs (ResNet18, EfficientNetV2S).
  • Loss Functions: Primarily Triplet Loss (to push features from different writers apart and pull features from the same writer together) and ArcFace Loss (for improved feature separation by enforcing angular margins).
  • Post-processing:
    • Pooling Aggregation: Mean pooling to combine patch-level features into a document-level embedding.
  • PCA Dimensional Reduction: To reduce feature dimensionality.

Datasets

The study uses three datasets:

  • ICDAR2013-WI: A standard writer identification dataset with well-processed writing lines. Used for baseline performance measurement and comparing architectures.
  • ICDAR2017-HistoricalWI: A historical writer identification dataset with document scans. Used for training and testing method comparisons.
  • ICDAR2019-HisIR: A challenging historical dataset focusing on document retrieval, featuring raw document scans with significant noise and degradation. This dataset is crucial for evaluating the effectiveness of pre-processing techniques like Text-AOI selection.

Key Insights and Ideas

  • End-to-end isn't always better: Many end-to-end deep learning models struggle to generalize in realistic HWI scenarios, particularly zero-shot settings. This is attributed to weak capture of low-level features, inconsistent patch representations, and sensitivity to noise.
  • Pre-processing is critical: Simple image resizing is insufficient for noisy historical documents. Combining document binarization with a carefully designed Text-AOI selection method is essential for isolating meaningful writing patterns and improving robustness.
  • Text-AOI selection is a game-changer: This technique effectively filters out noise and irrelevant regions, allowing random patch sampling to focus on distinctive writing characteristics, which is crucial for training deep learning models.
  • SwinV2 shines: Transformer-based architectures, specifically SwinV2, demonstrate superior performance over CNNs by capturing both local and global patterns in document patches.
  • ArcFace Loss for superior separation: While Triplet Loss is a good starting point, fine-tuning with ArcFace Loss significantly improves feature discriminability, leading to better generalization, especially in zero-shot scenarios. This is because ArcFace enforces stricter angular margins, creating more separable representations.
  • Simplicity can win: A well-configured pipeline (SwinV2-Base + ArcFace) can achieve state-of-the-art results without overly complex architectures or post-processing.

Key Results

  • The proposed end-to-end setup (SwinV2-Base backbone, Text-AOI selection pre-processing, and ArcFace loss for fine-tuning) achieved performance comparable to complex state-of-the-art methods on both standard WI and HWI datasets.
  • Specifically, on the ICDAR2017-HistoricalWI dataset, their "Ours" method (SwinV2 + ArcFace) achieved a Top-1 accuracy of 97.15%, P@2 of 96.08%, and mAP of 42.16%, outperforming previous methods.
  • PCA dimensional reduction generally did not significantly improve results, suggesting that the learned feature representations are already well-structured.
  • SwinV2-Base with cosine Triplet loss and L2 regularization showed the most consistent loss reduction during training.

Limitations and Future Work

  • Training Cost: Fine-tuning with ArcFace Loss, while effective, is more resource-intensive and slower than Triplet Loss training.
  • Future Work: The authors suggest exploring more efficient fine-tuning strategies to reduce training costs. They also anticipate that the availability of larger historical writer identification datasets will further enable advancements in this field.

Summary updated

Abstract

This paper investigates various factors that influence the performance of end - to-end deep learning approaches for historical writer identification (HWI), a task that remains challenging due to the diversity of handwriting styles, document degradation, and the limited number of labelled samples per writer. These conditions often make accurate recognition difficult, even for human experts. Traditional HWI methods typically rely on handcrafted image processing and clustering techniques, which tend to perform well on small and carefully curated datasets. In contrast, end-to-end pipelines aim to automate the process by learning features directly from document images. However, our experiments show that many of these models struggle to generalise in more realistic, document-level settings, especially under zero-shot scenarios where writers in the test set are not present in the training data. We explore different combinations of pre-processing methods, backbone architectures, and post-processing strategies, including text segmentation, patch sampling, and feature aggregation. The results suggest that most configurations perform poorly due to weak capture of low-level visual features, inconsistent patch representations, and high sensitivity to content noise. Still, we identify one end-to-end setup that achieves results comparable to the top-performing system, despite using a simpler design. These findings point to key challenges in building robust end-to-end systems and offer insight into design choices that improve performance in historical document writer identification.
BibTeX
@inproceedings{rasyidi2025beyond,
  author = {Hanif Rasyidi and Moshiur Farazi},
  title = {{Beyond the Pipeline: Analyzing Key Factors in End-to-End Deep Learning for Historical Writer Identification}},
  booktitle = {DSAA},
  year = {2025},
  url = {https://openreview.net/forum?id=Q5eP0ONC4Y},
}
Figure