Mining the Past: A Comparative Study of Classical and Neural Topic Models on Historical Newspaper Archives

Keerthana Murugaraj, Salima Lamsiyah, Marten During, Martin TheobaldNAACL 2025

TL;DR

This paper tackles the challenge of extracting meaningful themes from vast historical newspaper archives. By comparing traditional topic modeling methods (LDA, NMF) with newer neural approaches (Top2Vec, BERTopic), they find that BERTopic, especially with careful preprocessing, significantly outperforms others in topic coherence and diversity, offering a more robust solution for historical NLP tasks.

What Problem Does This Paper Address?

Historians and social scientists are increasingly turning to digitized newspaper archives for rich historical insights. However, the sheer volume of these archives makes manual analysis impractical. Topic modeling (TM) offers an automated way to discover hidden themes, but choosing the right TM method is crucial. Classical methods like Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) have been widely used, but they often struggle to capture the nuanced, contextual semantics present in historical texts. Newer neural-based TM methods, leveraging powerful language embeddings, show promise but are underutilized in historical research. This paper aims to bridge this gap by providing a comprehensive empirical comparison of classical and neural TM methods on a large historical newspaper dataset, evaluating their performance across various preprocessing strategies.

Methods

The study compares four topic modeling techniques:

  • Classical Models:
    • LDA (Latent Dirichlet Allocation): A probabilistic model that represents documents as mixtures of topics and topics as distributions of words.
  • NMF (Non-Negative Matrix Factorization): A matrix decomposition technique that factorizes a term-document matrix into two lower-dimensional matrices representing topics and their word distributions.
  • Neural-Based Models:
    • Top2Vec: Leverages pre-trained word and document embeddings to discover topics.
  • BERTopic: Utilizes transformer-based embeddings (like BERT) and clustering techniques to identify topics.

The researchers also investigated two distinct preprocessing strategies:

  • Type 1 (Extensive Preprocessing): Included lowercasing, stopword removal, filtering unwanted patterns, punctuation removal, and lemmatization. This was primarily used for classical models.
  • Type 2 (Minimal Preprocessing): Focused on removing only unwanted patterns and whitespace, preserving more of the original text structure. This was used for neural models to assess the impact of less aggressive preprocessing.

The evaluation metrics used were Topic Coherence (TC), measuring the semantic similarity of words within a topic, and Topic Diversity (TD), assessing the range of distinct topics generated.

Datasets

The study utilized a large dataset of over 148,000 historical newspaper articles spanning from 1955 to 2018. This dataset was segmented into four time periods: 1955-1970, 1971-1986, 1987-2002, and 2003-2018. The dataset was specifically curated around the themes of "nuclear power" and "nuclear safety." No new benchmark datasets were introduced; the focus was on applying existing methods to a real-world historical corpus.

Key Insights and Ideas

  • Preprocessing is Paramount: The study strongly highlights that the choice of preprocessing significantly impacts topic model performance, especially for neural methods. Extensive preprocessing (Type 1) consistently led to better topic coherence and diversity for BERTopic.
  • BERTopic's Superiority: BERTopic, particularly when combined with Type 1 preprocessing, consistently outperformed LDA, NMF, and Top2Vec across various metrics, demonstrating its robustness and adaptability for historical newspaper archives.
  • Neural Models' Contextual Advantage: Neural models, by leveraging contextual embeddings, are better equipped to capture nuanced semantic relationships compared to classical models, which rely on word co-occurrence.
  • Trade-offs in Classical Models: LDA tends to produce more diverse topics but at the cost of coherence, while NMF offers better coherence but struggles with diversity and can lead to redundancy, especially with a high number of topics. Both classical models are limited by the need to pre-define the number of topics.
  • Embedding Model Impact: For neural models like BERTopic and Top2Vec, the choice of embedding model (e.g., MPNet, DistilBERT, GTE_base) can influence performance, with GTE_base showing an advantage in handling longer texts without chunking.

Key Results

  • BERTopic with Type 1 Preprocessing: Achieved the highest Topic Coherence (TC) and Topic Diversity (TD) scores on the historical newspaper dataset, outperforming all other tested models.
  • Computational Efficiency: BERTopic demonstrated a good balance between performance and computational demands, making it a scalable choice for large datasets. Top2Vec was found to be computationally expensive.
  • Qualitative Analysis: While numerical results showed LDA excelling in diversity and NMF in coherence, BERTopic provided more coherent and diverse topics simultaneously. Qualitative examination of topics revealed BERTopic's ability to capture more specific and contextually relevant terms. For instance, BERTopic identified topics related to "Easter" and "council" that were missed by other models.

Limitations and Future Work

  • Limited Model Variants: The study focused on specific variants of LDA and NMF. Future work could explore other variants of these models.
  • LLM Embeddings: The authors plan to explore BERTopic with recent Large Language Model (LLM)-based embeddings to further enhance topic representation and clustering.
  • Dynamic Topic Modeling: Incorporating dynamic topic modeling to capture the evolution of topics over time is a key future direction, enabling a more nuanced understanding of temporal trends.
  • Alternative Chunking Strategies: Investigating other chunking strategies for neural models is also planned.

Summary updated

BibTeX
@inproceedings{murugaraj2025mining,
  author = {Keerthana Murugaraj and Salima Lamsiyah and Marten During and Martin Theobald},
  title = {{Mining the Past: A Comparative Study of Classical and Neural Topic Models on Historical Newspaper Archives}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure