TL;DR
Researchers have created a large, annotated dataset of historical New Zealand newspaper articles, featuring "soft" genre and topic labels with confidence scores. This novel dataset aims to improve the discoverability and analysis of historical texts by capturing the inherent subjectivity and hybridity of genre, paving the way for more nuanced computational understanding of historical media.
What Problem Does This Paper Address?
Digitized historical newspaper collections are vast and growing, offering immense research potential. However, their sheer scale and the complex, often ambiguous nature of their content pose significant challenges for researchers. Specifically, identifying and categorizing articles by genre (e.g., "news," "editorial," "advertisement") and topic (e.g., "politics," "sport") is difficult due to the fluid definitions of genres, the impact of OCR errors, and the subjective interpretation of content. Traditional classification methods often rely on single, "hard" labels, which can lose valuable information about the nuances and hybridity of historical texts. This paper addresses the need for a richer, more representative dataset that can better capture this complexity for advanced computational analysis.
Methods
The authors developed a custom annotation system using Google Colab and Google Cloud Storage. This system allowed for the display of article images, support for multiple genre and topic annotations per article, and the capture of annotator confidence scores. A key methodological innovation is the use of "soft labels" and confidence scores, aligning with a "perspectivist" approach to machine learning. This acknowledges that genre and topic can be subjective and that articles can embody multiple categories simultaneously. Annotators were trained on guidelines and worked collaboratively to discuss challenging cases and refine the process. Inter-annotator agreement was measured using Krippendorff's alpha.
Datasets
The primary dataset is a newly created, annotated collection of 7,036 historical newspaper articles from 106 New Zealand newspaper titles, spanning the period 1839-1903. This dataset was sampled from the National Library of New Zealand's Papers Past open data. The dataset includes:
- Soft genre labels: Multiple genre labels with associated confidence scores for each article.
- Topic labels: Up to four topic words per article, also with confidence scores.
- Annotator confidence scores: Indicating the annotator's certainty about their genre and topic assignments.
The dataset also includes "rough" genre labels derived from article titles, which were used for initial candidate article identification.
Key Insights and Ideas
- The "Perspectivist" Approach: Embracing soft labels and confidence scores is crucial for capturing the inherent subjectivity and hybridity of genre in historical texts, moving beyond rigid, single-label classifications.
- Genre Ambiguity is Informative: The dataset's design acknowledges that articles can legitimately belong to multiple genres, and this ambiguity itself is a valuable piece of information for analysis.
- The "Various" Label: The prevalence of the "various" topic label highlights the complexity and multifaceted nature of historical newspaper content, suggesting that simple topic extraction might miss nuanced themes.
- Inter-Annotator Agreement: While inter-annotator agreement (measured by Krippendorff's alpha) was moderate, it was considered good given the subjective nature of genre classification and the historical context. The consensus genre approach showed higher agreement than the primary genre alone.
Key Results
- A dataset of 7,036 articles was created, annotated by six annotators.
- 2,225 articles (32%) were at least double-annotated.
- The primary genre selection ("Genre 1") had 66.2% agreement among annotators for articles with at least two annotators.
- Krippendorff's alpha scores for the full dataset ranged from 0.66 to 0.86, indicating moderate to good agreement.
- The "Recipe" genre showed the highest per-genre agreement (alpha = 0.93), while "Opinion" had the lowest (alpha = 0.36).
- The top 20 topics revealed the significance of labels like "various" and "politics," with "various" being applied to nearly three times as many articles as "politics."
- The median confidence score for "Notice" was 1.0, despite having one of the lowest alpha scores, suggesting a high degree of certainty but also potential for disagreement on its precise categorization.
Limitations and Future Work
- Dataset Balance: The dataset is not yet balanced across all genres, with some newspapers being over-represented. Future work will involve sampling additional articles to boost lower-frequency genres.
- Duplicates: The dataset contains duplicates due to article reprints across different issues and titles. De-duplication methods will be explored.
- Topic Label Granularity: The free-text topic annotation resulted in a large, sparse set of labels. While flexible, a predefined list of topics was suggested by some annotators.
- Annotator Fatigue: The demanding nature of the annotation task led to annotator fatigue, potentially impacting consistency and accuracy.
- Impact of Confidence Scores: The authors plan to evaluate the impact of using confidence scores as weights or thresholds in future work.
- Interpretability of Models: The primary goal is to use this dataset to develop interpretable classification models that enable fine-grained exploration of historical newspapers.
- Cultural Sensitivity: The historical context of the articles, reflecting colonial perspectives and potentially outdated or harmful attitudes, will be carefully documented and considered in future use.
Summary updated