TL;DR
This paper introduces "GoldenHymns(S)," a novel dataset of 19th-century Danish religious hymns, complete with modernized Danish versions, English translations, and verse-level sentiment annotations. They use this dataset to evaluate various sentiment analysis tools, finding that traditional dictionary-based methods surprisingly outperform transformer models on this specific, historically nuanced text genre.
What Problem Does This Paper Address?
Computational linguistics often struggles with specialized literary domains, especially for less-resourced languages like Danish. Religious hymns, a significant part of Danish cultural heritage, present a unique challenge due to their archaic language, figurative expressions, and specific religious contexts. While sentiment analysis (SA) is a growing field in digital humanities, its application to historical Danish literature is limited. Existing Danish SA benchmarks are scarce and primarily focus on modern language. Furthermore, understanding how sentiment evolves over time within a literary tradition (diachronic trends) is crucial for a deeper cultural and historical analysis, but requires specialized datasets and tools. This paper aims to bridge this gap by creating a resource for studying sentiment shifts in historical Danish hymns and evaluating existing SA tools' performance on this challenging material.
Methods
The authors introduce the GoldenHymns(S) dataset, which comprises 65 Danish religious hymns from 1798-1873. To facilitate SA and cross-linguistic comparison, each hymn verse was:
- Modernized: Translated into contemporary Danish spelling and grammar.
- Translated: Rendered into English.
- Annotated for Valence: Two human experts assigned a sentiment score from 0 (lowest) to 10 (highest) to each verse, focusing on the embedded sentiment rather than broader context. Inter-rater agreement was measured at a Spearman's r of 0.726.
They then evaluated several SA tools on this dataset, categorizing them into:
- Dictionary-based tools: Afinn, Sentida, and Asent.
- Transformer-based models: Senda (Danish-specific), Alexandra Institute sentiment base (Danish-oriented), and RoBERTa xlm (multilingual).
The evaluation was conducted on both the original and modernized hymn texts to assess the impact of language change.
Datasets
- GoldenHymns(S): A newly created dataset of 65 Danish religious hymns (1798-1873), including original texts, modernized Danish versions, English translations, and human-annotated valence scores for each verse. This dataset is made publicly available.
- Existing Danish Hymnal Datasets: The paper references prior work on Danish hymnal collections, noting their limitations in facilitating diachronic study (e.g., focusing on single authors or limited periods).
Key Insights and Ideas
- Dictionary-based SA holds its own: Surprisingly, traditional dictionary-based SA tools (especially Sentida) performed as well as, or even better than, sophisticated transformer-based models on this historical religious text genre. This challenges the common assumption that transformer models universally outperform older methods on literary texts.
- Modernization matters: All SA tools showed improved performance when applied to the modernized Danish versions of the hymns, highlighting the significant impact of archaic language on computational analysis.
- Verse-level analysis is key: The paper's focus on verse-level sentiment annotation is crucial, as hymns often lack complete sentences within a single verse, making sentence-level analysis problematic.
- Historical context is vital for annotation: The authors emphasize the importance of using annotators familiar with the historical register and domain to achieve reliable sentiment scores, as seen in their relatively high inter-rater agreement.
Key Results
- Overall Performance: Dictionary-based tools (Afinn, Sentida, Asent) showed comparable or better performance (Spearman's rho) than transformer-based methods on the GoldenHymns(S) dataset.
- Impact of Modernization: All tested models showed improved performance on modernized texts compared to original texts. Sentida and RoBERTa xlm saw the biggest gains.
- Best Performing Tool: Sentida consistently performed best across all collections and versions of the hymns.
- Transformer Models: Among transformer models, the Alexandra Institute and RoBERTa xlm models performed best.
- Performance Over Time: Models generally improved over time when applied to hymn collections from later periods (1798, 1857, 1873), though this trend was slightly weaker for modernized texts.
Limitations and Future Work
- Dataset Size: The current "Small" (S) designation of the dataset reflects its composition of 65 hymns, with plans for expansion.
- Model Exclusions: The authors opted to exclude GPT models and other generative large language models due to their opacity and prompt-dependent variability, suggesting this as a future research direction.
- Future Directions:
- Expanding the dataset in terms of size and temporal range.
- Testing the impact of supplying lexica of historical language variants.
- Examining the development of hymnal sentiment across time and in relation to specific concepts.
- Fine-tuning models on this type of sentiment annotation.
- Investigating the behavior of generative large language models on this task.
Summary updated