TL;DR
This paper tackles the challenge of understanding how word emotions evolve over time by adapting existing sentiment analysis algorithms to a richer Valence-Arousal-Dominance (VAD) model. They introduce the first gold standard for historical word emotions and demonstrate that using larger, more general sets of seed words for training is more effective than relying on small, supposedly stable sets.
What Problem Does This Paper Address?
Understanding historical texts requires grasping how language, including the emotional connotations of words, changes over time. While previous work has often focused on simple positive/negative sentiment (polarity), this paper aims for a more nuanced understanding using the Valence-Arousal-Dominance (VAD) model. Existing methods for inducing word emotions are often not well-suited for historical analysis, and there's a lack of robust evaluation benchmarks for this task. A key claim in prior research is that using small sets of "stable" seed words is crucial for historical sentiment analysis, a claim this paper investigates.
Methods
The authors adapt two popular word polarity induction algorithms (PARASIM and RANDOM WALK) to work with the VAD model, allowing for a more fine-grained analysis of emotion. They also utilize a k-Nearest-Neighbor (KNN) algorithm that already supports vectorial input. For word similarity, they employ cosine similarity on word embeddings, a standard technique in distributional semantics.
Datasets
- Seed Lexicons:
- English: Extended version of ANEW (Affective Norms for English Words) by Warriner et al. (2013), containing ~14k entries.
- German: Affective Norms for German Sentiment Terms (ANGST) by Schmidtke et al. (2014), containing 1,003 words.
- Historical Corpora for Embedding Training:
- Corpus of Historical American English (COHA)
- Deutsches Text Archiv (DTA)
- Newly Introduced Benchmark:
- A gold standard of historical word emotions for both English and German, consisting of 100 high-frequency words each. This dataset was annotated by doctoral students experienced in 19th-century texts, who were asked to adopt the perspective of speakers from that era. This differs from existing lexicons by focusing on historical language stages.
Key Insights and Ideas
- Quantity over Supposed Stability: The most striking finding is that using larger, more comprehensive seed word sets consistently outperforms smaller, hand-selected sets of "temporally stable" words, even if some words in the larger set might not be perfectly accurate for the target historical period. This directly challenges previous assumptions.
- VAD Model's Advantage: Adapting algorithms to the VAD model provides a richer understanding of emotion compared to simple polarity, enabling more nuanced diachronic analysis.
- Expert Annotators Matter: The newly created gold standard, annotated by historical language experts, shows higher inter-annotator agreement than contemporary lexicons created via crowdsourcing, suggesting experts can achieve greater consensus even for historical language.
- Embedding Choice Matters: The choice of word embedding algorithm (SVDPPMI vs. SGNS) can significantly impact performance, with SVDPPMI showing an advantage for the full seed lexicon in some cases.
Key Results
- Synchronic Evaluation:
- For the full seed lexicon, PARASIMNUM with SVDPPMI embeddings achieved the highest correlation (r ≈ 0.679 for Valence, 0.445 for Arousal, 0.547 for Dominance).
- For the limited seed lexicon, RANDOMWALKNUM performed significantly better than other approaches, but overall results were much weaker than with the full seed lexicon.
- Diachronic Evaluation:
- For English, using full seed lexicons yielded performance around r = .35. Using limited seed lexicons resulted in markedly weaker performance across all conditions.
- German results with full seed lexicons were comparable to English, with SVDPPMI outperforming SGNS.
- The paper found that the hand-selected "stable" seed words did not improve performance and, in most cases, were harmful.
Limitations and Future Work
- Gold Standard Size: The authors acknowledge that their historical gold standard is limited in size (100 words per language). To achieve statistically significant differences (p < 0.05) in correlation, they estimate the size would need to be increased by a factor of about 40, which is challenging due to the reliance on expert annotators.
- Data Sparsity: The paper notes that data sparsity remains a challenge when working with historical language.
- Future Work: The authors plan to continue working on solutions to address data sparsity and explore more advanced machine learning approaches for historical language analysis. They also suggest that the combination of SVDPPMI embeddings with PARASIMNUM is a robust choice for future work due to its stable performance and minimal hyperparameter tuning.
Summary updated