TL;DR
This paper introduces RAPA, a novel framework for sentiment analysis in historical texts. RAPA tackles the challenge of scarce and noisy data by using Large Language Models (LLMs) with "role-guided" prompts to simulate historical perspectives for annotation, and then employs a prototype-aligned representation learning approach to denoise these labels and improve model robustness.
What Problem Does This Paper Address?
Analyzing sentiment in historical literature offers a window into past societies and their worldviews. However, this task is significantly hampered by two main issues:
- Data Scarcity: Annotated historical texts are rare, requiring specialized linguistic and historical expertise to create.
- Model Generalization: Models trained on modern text often fail to capture the nuanced, implicit, ironic, or morally complex sentiment expressions common in historical literature.
Existing approaches often rely on sentiment lexicons (which are scarce for historical texts) or use LLMs for annotation. While LLMs show promise, their direct prompting often leads to labels reflecting modern sentiment norms, missing historical context and introducing noise. This paper aims to overcome these limitations by developing a more context-aware and robust annotation and learning strategy.
Methods
RAPA consists of two core components:
- Role-Guided Annotation Strategy: Instead of generic prompts, this strategy instructs LLMs (specifically ChatGPT-4o) to adopt the persona of a historical figure or a culturally situated reader. This "role-play" encourages the LLM to interpret sentiment through the lens of the historical period, considering its specific linguistic norms, values, and rhetorical strategies. Constraints are added to the prompts to explicitly limit the LLM's knowledge to the historical context, mitigating modern biases.
- Prototype-Aligned Representation Learning: This is a two-stage framework designed to denoise potentially noisy LLM-generated labels and improve representation robustness.
- Stage 1: Learning High-Resource Prototypes: Sentiment prototypes (class-level representations) are learned from abundant, high-quality "high-resource" datasets. These prototypes act as stable semantic anchors.
- Stage 2: Prototype Alignment and Contrastive Adaptation: The framework aligns representations from low-resource datasets (with limited or noisy labels) to these high-resource prototypes. This is achieved through a joint loss function that combines:
- Cross-Entropy Loss: Standard sentiment classification loss on the low-resource data.
- Symmetric Contrastive Loss: This novel loss function encourages alignment between low-resource and high-resource prototypes in a shared semantic space. It helps the model learn robust representations by reducing reliance on potentially noisy individual labels, leveraging the semantic consistency of prototypes.
Datasets
The study utilizes a combination of high-resource and low-resource datasets:
- High-Resource Datasets (for learning prototypes):
- SemEval 2017 Task 4 (English, Spanish, Arabic)
- IMDB (English)
- GoEmotions (Subset, English) - repurposed for "mixed" sentiment.
- Low-Resource Datasets (for fine-tuning and evaluation):
- Lessing's Plays (German, 18th century drama)
- Scandinavian Novels (German, 19th century fiction)
- Austrian Newspapers (German, 19th century news)
- EvaLatin 2024 (Latin, 1st-19th century poetry)
The authors note that for Lessing's Play and EvaLatin 2024, they augmented the datasets by including additional plays and poems, respectively, and used ChatGPT for low-resource sentiment adaptation.
Key Insights and Ideas
- Historical Context is Crucial for LLM Annotation: Simply asking LLMs to classify sentiment is insufficient for historical texts. Simulating historical perspectives through role-playing prompts significantly improves the quality and relevance of generated labels.
- Prototypes as Semantic Anchors: High-resource sentiment prototypes provide stable semantic anchors that can guide the learning process for low-resource datasets, even when those datasets have noisy labels.
- Representation-Level Denoising: Instead of just cleaning labels, aligning representations to prototypes offers a more robust way to handle noise and improve generalization across domains.
- Bridging the Gap: The symmetric contrastive loss effectively bridges the representational gap between high-resource and low-resource domains, enabling knowledge transfer.
Key Results
- Outperforms Baselines: RAPA consistently outperforms state-of-the-art baselines and direct fine-tuning methods across multiple historical sentiment datasets, achieving the best performance on 3 out of 4 datasets.
- Improved LLM Annotation: Role-guided annotation improved performance over generic LLM prompts, particularly on the Latin corpora (up to +17% gain).
- Robustness to Noise: RAPA demonstrates strong robustness to both machine-generated (LLM) and human-induced annotation noise. For example, it improved macro-F1 by +12.32 on Austrian News with InfoXLM-large.
- Prototype Alignment Benefits: Ablation studies showed that removing high-resource prototype alignment led to consistent performance degradation, highlighting its importance for stabilizing representations and improving generalization. Trainable high-resource prototypes also outperformed fixed strategies.
Limitations and Future Work
- Manual Prompt Design: The role-guided annotation strategy currently requires manual design of customized prompts for different datasets, limiting full automation.
- Limited Scope: The study focused on a limited set of historical languages and genres. Generalizing RAPA to broader historical corpora and underrepresented languages is a promising future direction.
- Modular Design: The current framework separates cue design (prompts) from model training. Future work could explore jointly optimizing annotation cues and representation learning for enhanced adaptability.
- Potential for Bias: The use of LLMs for annotation, even with role-playing, may introduce biases, especially when interpreting culturally sensitive content. The generated labels do not reflect the authors' views.
Summary updated