Abstract
Shrey Desai, Barea Sinno, Alex Rosenfeld, Junyi Jessy Li. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Political scientists frequently analyze historical archives to track the evolution of policy and discourse over decades. However, massive datasets like the Corpus of Historical American English (COHA) lack topic labels, creating a "needle in a haystack" problem. While modern labeled datasets exist (e.g., the New York Times), models trained on them often fail when applied to historical text due to diachronic shift—changes in vocabulary, style, and naming conventions over time. Standard unsupervised methods like Latent Dirichlet Allocation (LDA) often yield vague categories when applied to such noisy, large-scale data.
To address this, researchers at UT Austin developed Adaptive Ensembling (AE), a framework designed to adapt models trained on modern labeled data to unlabeled historical archives.
Methodology: Adaptive Ensembling
The AE framework builds upon the "Mean Teacher" self-ensembling approach. In this architecture, a "student" network is trained on labeled source data, while a "teacher" network—a moving average of the student—provides consistent targets for the student on unlabeled target data.
The authors introduce three primary innovations to improve performance on complex textual data:
1. Adaptive Constants
Standard self-ensembling uses a fixed smoothing factor (α) for teacher updates. AE replaces this with trainable, parameter-specific "adaptive constants." This allows the model to fine-tune the rate at which different features are learned by the teacher, stabilizing the training process on the often-volatile loss surfaces of text data. Visualizations reveal that different network layers adapt at varying rates, a nuance missed by fixed hyperparameters.
2. Temporal Curriculum
Language change is gradual. The authors observed that vocabulary overlap is significantly higher between modern text and recent historical documents than with older ones.
Based on this, they implement a "temporal curriculum": the model is first trained on the most recent historical data and progressively moves backward in time, using the easier, more similar data as a bridge to more archaic language.
3. Specialized Architecture
The model utilizes a 1D Convolutional Neural Network (CNN) with "sequence squeezing" to compress text into dense representations. It also incorporates Time Embeddings, allowing the network to explicitly account for the publication year of a document.
Experimental Setup
- Source Domain: New York Times Annotated Corpus (1987+), labeled for political topics.
- Target Domain: COHA (1922–1986), unlabeled.
- Evaluation: A new benchmark of 350 expert-annotated COHA documents was created to rigorously test the models on historical data.
Key Results
Adaptive Ensembling demonstrated significant improvements over both "Source Only" baselines and established domain adaptation methods like Domain-Adversarial Neural Networks (DANN):
- Binary Classification: AE achieved a 77.1 macro-F1 score, nearly doubling the 46.2 score of the source-only model and outperforming DANN (65.8).
- Multi-Label Classification: AE reached 61.1 macro-F1, significantly outperforming the 39.6 score of the source-only baseline.
- Stability: AE exhibited much more stable loss curves during training compared to adversarial methods, which are prone to oscillation.
Beyond metrics, the model's practical utility was confirmed by running LDA on the documents it identified as political. This "cleaned haystack" produced sharp, historically relevant topics (e.g., "Vietnam conflict") that were lost in the noise of the full corpus.
Conclusion
Adaptive Ensembling effectively bridges the temporal gap in political document analysis. While it requires publication metadata for its curriculum, it significantly reduces the need for manual hyperparameter tuning through its adaptive constants. Future research may extend this framework to non-temporal domain shifts, such as adapting between distinct writing styles or genres.
Summary updated