Historical Ink: Exploring Large Language Models for Irony Detection in 19th-Century Spanish

Kevin Cohen, Laura Manrique-Gómez, Ruben ManriqueNAACL 2025

TL;DR

This paper explores using Large Language Models (LLMs) like BERT and GPT-4o to detect irony in 19th-century Spanish newspapers, a challenging task due to historical linguistic nuances. The authors introduce a novel semi-automated annotation methodology, combining LLM capabilities with human expertise, to create a more robust dataset and improve irony detection accuracy.

What Problem Does This Paper Address?

Detecting irony in text is notoriously difficult for Natural Language Processing (NLP) systems. This challenge is amplified when dealing with historical texts, where language, cultural references, and societal norms differ significantly from modern times. 19th-century Spanish newspapers, in particular, present a unique linguistic landscape with subtle expressions of irony that are hard for current models to grasp. Existing approaches often struggle with the nuances of historical language, class imbalance in datasets, and stylistic biases. This research aims to bridge this gap by developing and evaluating methods for more accurate irony detection in this specific historical context.

Methods

The study employs a two-pronged approach: dataset enhancement and a BERT-based classification pipeline.

  1. Dataset Enhancement:
    • Prompt-based Enhancement: GPT-4o was used with carefully crafted prompts to expand existing text entries, aiming to emphasize emotional and contextual cues without altering the original meaning. This created an "ENHANCED" dataset.
  2. Semi-Automated Annotation: A more sophisticated approach involved using GPT-4o to initially tag a large corpus of previously untagged 19th-century Spanish newspaper texts. Crucially, human experts then reviewed these LLM-generated tags and justifications, correcting errors and ensuring historical authenticity. This process led to an "AUGMENTED" dataset, which was more balanced and contained higher-quality annotations.
  3. BERT-based Classification Pipeline:
    • A standard BERT encoder architecture with a feed-forward neural network head was used for classification.
  4. Various BERT models were tested, including general-purpose, multilingual, and Spanish-specific variants, as well as a model specifically fine-tuned on 19th-century Spanish texts.
  5. The pipeline was evaluated for both multi-class (irony, positive, negative, neutral) and binary (irony vs. not irony) classification tasks.

Datasets

  • LatamXIX Dataset: The primary source of text samples, consisting of 19th-century Latin American newspapers.
  • PRIMARY Dataset: An initial manually annotated subset of the LatamXIX Dataset (2,734 entries) with four classes: "IRONÍA" (irony), "POSITIVO" (positive), "NEGATIVO" (negative), and "NEUTRO" (neutral).
  • ENHANCED Dataset: Created by applying GPT-4o's prompt-based text expansion to the PRIMARY dataset.
  • AUGMENTED Dataset: A new, larger dataset (3,750 entries) created through the semi-automated annotation process, which significantly improved the balance of the "IRONÍA" class.

Key Insights and Ideas

  • Human-in-the-Loop is Crucial: While LLMs like GPT-4o can generate initial annotations, human expertise is indispensable for verifying accuracy, correcting subtle errors, and ensuring the preservation of historical linguistic authenticity.
  • Context is King (Especially Historical Context): Simply enhancing emotional content with LLMs wasn't enough to significantly boost irony detection. The specific cultural and historical context of the 19th century proved to be a major hurdle, highlighting potential biases in general-purpose LLMs.
  • Semi-Automated Annotation is a Game Changer: The semi-automated approach, combining LLM efficiency with human oversight, was highly effective in expanding and balancing the dataset, particularly for the underrepresented "IRONÍA" class.
  • Binary Classification Simplifies: The task of classifying text as simply "ironic" or "not ironic" generally yielded better performance than multi-class classification, suggesting that simplifying the problem space aids the models.

Key Results

  • Baseline GPT-4o Performance: Relying solely on GPT-4o for classification yielded the weakest performance, with a multi-class accuracy of only 0.39.
  • BERT-based Pipeline Improvement: The BERT-based pipeline significantly outperformed GPT-4o, achieving an accuracy of 0.66 for multi-class and 0.72 for binary classification on the PRIMARY dataset.
  • Augmented Dataset Dominance: The AUGMENTED dataset, created through semi-automated annotation, led to the best results. In binary classification, the dccuchile/bert-base-spanish-wwm-cased model achieved an F1 score for the 'IRONY' class of 0.77, a substantial improvement.
  • Recall Improvement: The semi-automated process significantly improved recall for the 'IRONY' class, indicating the model could detect irony in more instances.
  • Human vs. LLM Tagging Discrepancy: Human evaluators tagged 53.1% of entries as ironic, while GPT-4o tagged 73.6%. Humans also identified more negative and positive sentiments that GPT-40 missed, suggesting LLMs can struggle with nuanced sentiment and historical context.

Limitations and Future Work

  • Reliance on GPT-4o: The initial dataset enhancement steps relied on GPT-4o, which is costly and limits scalability. Future work could explore more accessible and cost-effective LLMs or alternative augmentation strategies.
  • Historical Nuances: Despite advancements, capturing the full depth of historical irony remains challenging. Future research should focus on refining domain-specific prompts, exploring more advanced LLM architectures, and developing agent-based workflows that systematically incorporate historical context.
  • Broader Datasets: Expanding and enriching historical datasets across diverse cultural and historical contexts is crucial for developing more reliable and generalized irony detection models.

Summary updated

BibTeX
@inproceedings{cohen2025historical,
  author = {Kevin Cohen and Laura Manrique-Gómez and Ruben Manrique},
  title = {{Historical Ink: Exploring Large Language Models for Irony Detection in 19th-Century Spanish}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure