Can GPT4 Detect Euphemisms across Multiple Languages?

Todd Firsich, Anthony RiosNAACL 2024

TL;DR

This paper explores GPT-4's ability to detect euphemisms across multiple languages using prompt engineering, achieving a top-ranking performance in a shared task. While GPT-4 shows promise, its performance varies significantly across languages, highlighting the challenges of cross-lingual figurative language understanding.

What Problem Does This Paper Address?

Euphemisms, words or phrases used to soften harsh or offensive language, are crucial for nuanced communication. However, their inherent ambiguity and cultural variability make them a significant challenge for Natural Language Processing (NLP) systems. While previous research has focused on fine-tuning transformer models for euphemism detection, this paper investigates the potential of large language models (LLMs) like GPT-4, specifically using prompt engineering, to tackle this task across multiple languages. The goal is to understand how effectively GPT-4 can identify euphemisms without task-specific fine-tuning, especially in a multilingual context.

Methods

The authors employ a few-shot prompting framework to evaluate GPT-4's euphemism detection capabilities. This involves providing GPT-4 with a prompt that includes:

  • Instruction: A clear directive on the task.
  • Context: A definition of what a euphemism is.
  • Euphemism Examples: Instances of phrases used euphemistically.
  • Literal Examples: Instances of phrases used literally.
  • Task: The specific sentence and phrase to be classified.

They explore several prompting strategies:

  • Zero-Shot: Only instruction and task.
  • Zero-Shot with Context: Adds the definition of euphemism.
  • Few-Shot with Random Examples: Includes one random euphemism and one literal example.
  • Few-Shot with Targeted Examples: Includes k euphemism and k literal examples that specifically use the same phrase (PET) as the test instance. This approach aims to leverage more relevant in-context learning.

The experiments were conducted using the OpenAI API with the gpt-4-0125-preview model, setting the temperature to 0 for deterministic outputs.

Datasets

The study utilizes data from the 2024 FigLang shared task on Multilingual Euphemism Detection. This dataset includes samples in Chinese, English, Spanish, and Yorùbá, with both training and testing sets. The training sets contain sentences, the potential euphemistic term (PET), and a classification label (euphemism or not). The test sets provide sentences and PETs without ground truth labels. The authors note varying degrees of overlap between PETs in the training and test datasets across languages.

Key Insights and Ideas

  • Prompting as a Powerful Alternative: Prompt engineering with GPT-4 offers a viable and resource-efficient alternative to traditional fine-tuning for euphemism detection, especially for tasks requiring nuanced language understanding.
  • Context is King (Even for LLMs): Simply providing the definition of a euphemism (Zero-Shot with Context) significantly improves performance over a basic Zero-Shot approach, underscoring the importance of contextual information for LLMs.
  • Targeted Examples Outperform Random: Using specific, relevant examples in few-shot prompts (Few-Shot with Targeted Examples) consistently yields better results than using randomly selected examples. This highlights the LLM's ability to learn from closely related instances.
  • Language Matters (A Lot): GPT-4's performance is not uniform across languages. The significant performance gap between English and Spanish suggests that cultural nuances and linguistic differences pose substantial challenges for cross-lingual euphemism detection, even for advanced LLMs.
  • Data Overlap is Crucial: The lack of overlap between training and test PETs for Spanish appears to be a major factor in its poor performance, suggesting that LLMs benefit greatly from seeing similar examples during training.

Key Results

The authors' best system, using "Few-Shot - Targeted Examples (8)," achieved an average macro F1 score of .732 on the test dataset, ranking first in the competition.

  • English: Achieved the highest F1 score of .831.
  • Spanish: Showed the lowest performance with an F1 score of .598, a difference of .233 from English.
  • Chinese and Yorùbá: Performed in between English and Spanish.
  • The "Few-Shot - Targeted Examples" approach consistently outperformed other prompting methods, with performance increasing as more examples were provided (e.g., from .801 with 4 examples to .859 with 8 examples for Chinese in training).

Limitations and Future Work

  • Language-Specific Performance Gaps: The significant performance disparity across languages, particularly the low score for Spanish, is a key limitation. This is partly attributed to the lack of overlap between training and test data for Spanish PETs.
  • Contextual Ambiguity: For terms like "disabled," which can be used euphemistically or literally depending on context, GPT-4 struggled without strong examples, indicating a need for more sophisticated contextual understanding.
  • Future Directions:
    • Investigating LLM-generated samples to improve targeted prompts.
  • Developing better strategies for selecting more relevant targeted examples.
  • Exploring methods for LLMs to identify potential euphemisms without explicit prompting or pre-defined PETs.
  • Evaluating a wider range of LLMs beyond GPT-4.
  • Further research into how example ordering in prompts affects performance.

Summary updated

BibTeX
@inproceedings{firsich2024can,
  author = {Todd Firsich and Anthony Rios},
  title = {{Can GPT4 Detect Euphemisms across Multiple Languages?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure