TL;DR
This paper introduces News Déjà Vu, a novel semantic search tool that uses transformer language models to find historical news articles semantically similar to modern queries. This allows social scientists and the public to explore historical precedents for contemporary phenomena, overcoming the challenges of vast, noisy historical text archives.
What Problem Does This Paper Address?
Social scientists and the general public often seek to understand current events by drawing parallels with the past. However, historical texts, particularly digitized newspapers, are often vast, unstructured, and riddled with OCR noise, making traditional keyword searches brittle and ineffective. This makes it difficult to uncover relevant historical context for contemporary issues, hindering a deeper understanding of societal trends and phenomena.
Methods
The core of News Déjà Vu is a contrastively trained bi-encoder model that leverages transformer large language models. Before semantic search, a named entity recognition (NER) model is applied to mask out specific entities (people, locations, organizations). This crucial step allows the model to focus on broader thematic parallels rather than getting sidetracked by specific names that might differ across time periods. The bi-encoder then maps modern queries and historical articles into a shared embedding space, enabling efficient retrieval of semantically similar historical documents. The authors specifically tuned their NER model for noisy historical texts and used a contrastive training approach to mitigate the "anisotropy" of BERT-like models, improving the quality of document representations.
Datasets
- Training Data: The authors created their own training data by pairing modern news articles with historical ones. This involved using articles from the "American Stories" dataset (a Hugging Face dataset containing over 430 million historical public domain newspaper articles) and the Allsides news aggregator. They also used a large corpus of off-copyright newspaper articles from 1922-1977 for NER model training.
- Evaluation Data: The evaluation of the bi-encoder model was done by a skilled annotator who classified 350 randomly selected modern articles and their retrieved historical neighbors.
Key Insights and Ideas
- The Power of Entity Masking: Masking named entities is a critical step for finding thematic parallels across different time periods. It shifts the focus from specific actors to the underlying narrative and societal dynamics.
- Bridging the Gap for Social Scientists: The tool is designed to be user-friendly for social scientists who may not have extensive deep learning expertise, democratizing access to historical text analysis.
- Addressing OCR Noise: The NER model was specifically tuned to handle the noise inherent in historical OCR, a common challenge in digital humanities research.
- Contrastive Training for Semantic Similarity: The paper highlights the effectiveness of contrastive training in improving semantic similarity applications, particularly for mitigating issues with embedding space anisotropy.
Key Results
- The custom NER model achieved an F1 score of 90.4, outperforming a Roberta-Large model fine-tuned on CoNLL03.
- The News Déjà Vu bi-encoder model achieved a pairwise F1 of 92.4 on the test set, significantly outperforming existing models like those from Reimers and Gurevych (2019) and Silcock et al. (2022).
- Qualitatively, in 85.7% of cases, the retrieved historical articles shared the same major topic as the modern query, demonstrating the effectiveness of the semantic search.
Limitations and Future Work
- Language Support: Currently, News Déjà Vu only supports English. The authors suggest that extending it to other languages could be relatively straightforward by starting with multilingual Sentence-BERT weights.
- Human Expertise Still Crucial: The authors emphasize that while News Déjà Vu is a powerful tool for exploration, human expertise remains essential for drawing deeper insights and placing parallels in appropriate historical context.
- Potential for Misinterpretation: The tool retrieves articles based on similar language, which may not always reflect true underlying similarities in events or phenomena. Human judgment is needed to interpret these findings.
- Future Work: The authors suggest that the package could be extended to support more languages and that users can fine-tune their own models using their pre-trained weights.
Summary updated