Abstract
Insightful findings in political science often require researchers to analyze documents of a certain subject or type, yet these documents are usually contained in large corpora that do not distinguish between pertinent and non-pertinent documents. In contrast, we can find corpora that label relevant documents but have limitations (e.g., from a single source or era), preventing their use for political science research. To bridge this gap, we present adaptive ensembling, an unsupervised domain adaptation framework, equipped with a novel text classification model and time-aware training to ensure our methods work well with diachronic corpora. Experiments on an expert-annotated dataset show that our framework outperforms strong benchmarks. Further analysis indicates that our methods are more stable, learn better representations, and extract cleaner corpora for fine-grained analysis.
Introduction
Political scientists frequently seek to track specific themes, such as "international relations" or "political economy," across historical archives. However, they face a significant "domain gap" challenge. While labeled data is often available from modern sources like the New York Times (NYT), target data in archives like the Corpus of Historical American English (COHA) is unlabeled and spans decades or centuries. Language shifts over time—a political article from 1920 differs markedly in vocabulary and style from one in 2020. Standard tools like Latent Dirichlet Allocation (LDA) often fail in these scenarios, producing vague topics that conflate political and non-political terms. To address this, we introduce Adaptive Ensembling, an unsupervised domain adaptation framework designed to identify political content in unlabeled, historical text.
Methods
Our framework is built upon the "Mean Teacher" self-ensembling approach, where a "student" network learns from labeled source data and a "teacher" network (an exponential moving average of the student) provides stable targets for unlabeled data. We introduce three primary innovations:
- Adaptive Ensembling: Standard self-ensembling uses a fixed hyperparameter ($\alpha$) to update the teacher. We argue that different neurons learn at different rates and replace $\alpha$ with a matrix of trainable adaptive constants. This allows the model to fine-tune the teacher's reliance on the student for every parameter.
- Temporal Curriculum: To manage diachronic language shifts, we employ a curriculum that starts with target documents closest in time to the source data (where vocabulary overlap is highest) and progressively moves toward the distant past.
- Custom 1D-CNN Architecture: We utilize a 1D Convolutional Neural Network optimized for text, featuring sequence squeezing for compression, state connections for hierarchical features, and time embeddings to incorporate the document's era into the model's reasoning.
Datasets
We evaluate our framework using the New York Times Annotated Corpus (NYT) as the source domain (4,800 political and 4,800 non-political articles) and the Corpus of Historical American English (COHA) (1922–1986) as the target domain. To rigorously assess performance, we introduced a new expert-annotated test set for COHA, where political science graduate students labeled 350 documents for both binary and multi-label classification tasks.
Results and Analysis
Adaptive Ensembling (AE) significantly outperforms strong benchmarks, including Domain-Adversarial Neural Networks (DANN) and standard self-ensembling (SE). On the binary classification task, AE achieved a macro-F1 score of 77.1, an 11.4-point improvement over standard SE. In multi-label tasks, it reached a 61.1 macro-F1 score, a 10.1-point gain.
Beyond raw metrics, our analysis reveals that adaptive constants stabilize the training process. In standard models, the unsupervised loss often oscillates; however, in our framework, it converges smoothly, facilitating better feature alignment between domains.
Qualitative analysis further supports these findings. When using AE to filter COHA before running LDA, the resulting topics were highly specific (e.g., Vietnam conflict), whereas LDA on the raw, unfiltered corpus produced generic word clusters.
Conclusion
Adaptive Ensembling provides a robust solution for diachronic domain adaptation in political science. While the model currently relies on document metadata (publication year) for its temporal curriculum and embeddings, it demonstrates the power of layer-wise adaptive dynamics in complex text tasks. Future work will explore applying this adaptive approach to Transformer architectures and other social science domains.
Summary updated