Abstract
Determining the publication date of historical documents is a complex task in natural language processing, often hindered by document length and linguistic evolution. This paper presents a hierarchical deep learning approach developed for the DaDoEval task at EVALITA 2020. By combining SBERT sentence embeddings with a "bag-of-entities" representation, the model effectively captures both stylistic and contextual temporal cues. Experimental results on the writings of Italian statesman Alcide De Gasperi demonstrate that this approach significantly outperforms standard baselines across multiple time granularities: historical periods, five-year spans, and single years.
Introduction
Document dating—the task of predicting when a text was written—shares similarities with author identification and topic classification. However, it presents unique challenges. First, historical documents often exceed the 512-token limit of modern transformer models like BERT, and simple truncation risks losing critical temporal information. Second, language evolves; words and phrases change in meaning and usage over time, a phenomenon known as polysemy.
This study addresses these challenges within the context of the DaDoEval competition, focusing on the extensive written works of Alcide De Gasperi. The goal is to develop a robust system capable of dating documents at three levels of granularity: broad historical periods, five-year windows, and specific years.
Methodology
The proposed system utilizes a hierarchical architecture designed to process long documents without loss of context.
Sentence Embeddings and Entities
Instead of processing raw tokens, the model converts each sentence into a dense vector. After evaluating several models, including Universal Sentence Encoder (USE) and Language-agnostic BERT (LaBSE), Sentence-BERT (SBERT) was selected for its superior performance in this task.
To complement the stylistic information captured by embeddings, the model incorporates a Bag-of-Entities (BoE). Using the spaCy library's Named Entity Recognizer, the system identifies people and places mentioned in the text. These entities serve as "temporal anchors," providing specific historical context that general language models might miss.
Model Architecture
The sequence of sentence embeddings is fed into a Transformer block, followed by average pooling. This result is concatenated with the BoE vector and passed through a multi-layer dense classifier. The model is trained to simultaneously predict three temporal outputs.
Dataset and Task Definition
The primary dataset consists of 2,759 manually labeled documents written by De Gasperi between 1901 and 1954. The task is divided into three subtasks based on temporal granularity. The "Historical Period" subtask uses five distinct eras defined by political and social shifts in Italian history.
Evaluation was performed on a "same-genre" test set (similar to the training data) and a "cross-genre" test set consisting of personal letters (epistolary archives) to test the model's generalizability.
Key Results
The hierarchical model showed substantial improvements over the official task baselines, which utilized logistic regression and tf-idf.
| Subtask | Baseline (F1) | Model Result (F1) |
|---|---|---|
| Historical Period | 0.827 | 0.857 |
| Five-year Span | 0.485 | 0.609 |
| Single-year | 0.126 | 0.265 |
While the F1 score for single-year dating remains relatively low at 0.265, it represents a more than 100% improvement over the baseline, highlighting the difficulty of fine-grained temporal classification.
Discussion and Insights
Several key findings emerged from this research:
- The Failure of Truncation: Models that truncated documents to the first 512 tokens failed to converge, confirming that temporal clues are distributed throughout the entire text.
- Entities as Anchors: The Bag-of-Entities approach was vital for fine-grained dating. While embeddings capture the general stylistic trends of an era, entities like specific politicians or treaties anchor a text to a specific year.
- The Genre Gap: Performance dropped significantly on the cross-genre epistolary dataset. This suggests that document dating models are highly sensitive to writing style; a model trained on formal political discourse may struggle with informal personal correspondence.
Future work will focus on domain adaptation techniques to bridge this genre gap and exploring more complex architectures as hardware constraints allow.
Summary updated