Do LLMs Agree with Humans on Emotional Associations to Nonsense Words?

Yui Miyakawa, Chihaya Matsuhira, Hirotaka Kato, Takatsugu Hirayama, Takahiro Komamizu, Ichiro IdeACL 2024

TL;DR

This paper investigates whether Large Language Models (LLMs), specifically GPT-4, can accurately predict human emotional associations with nonsense words. The study found a notable positive correlation between GPT-4's predictions and human annotations, suggesting LLMs can serve as valuable tools for understanding human perception of novel words.

What Problem Does This Paper Address?

Creating effective product names, character designs, or brand identities often relies on evoking specific emotions through the sounds and spellings of words. However, understanding these human associations, especially with novel or "nonsense" words, is crucial but challenging due to the cost and labor involved in human studies. Previous research has explored using LLMs for estimating human perception of nonwords, but the emotional aspect, a core component of language meaning, remained largely unexamined. This paper aims to bridge this gap by quantifying the agreement between LLM-generated emotional associations for nonsense words and those provided by human annotators.

Methods

The researchers adapted a "best-worst-scaling" annotation procedure, originally used by Sabbatino et al. (2022) for human annotators, to evaluate GPT-4. For each target emotion (joy, sadness, anger, disgust, fear, surprise), GPT-4 was presented with a tuple of four nonsense words and asked to identify the word it associated most and least with that emotion. To ensure consistency and comparability with human data, the LLM was instructed to act as a native English speaker and to provide its reasoning step-by-step. The emotion intensity scores were then calculated using a formula based on the frequency of words being chosen as "most" or "least" associated, similar to the human annotation method.

Datasets

The study utilized a dataset of 272 nonsense words and 68 real words from Sabbatino et al. (2022). The real words were primarily used for comparison and attention checks in the original human study. For this LLM evaluation, only the 272 nonsense words were used to generate approximately 1,632 tuples for the LLM prompts.

Key Insights and Ideas

  • LLMs as Proxies for Human Perception: The core idea is that LLMs, with their vast training data, might have learned implicit associations between sounds, spellings, and emotions that mirror human intuition.
  • Beyond Simple Association: GPT-4's reasoning process revealed it doesn't just rely on phonetic similarity to real words but also considers sound symbolism (e.g., soft sounds for positive emotions) and the semantic implications of similar-sounding real words.
  • The "Bouba/Kiki" Effect in LLMs: The study implicitly tests if LLMs can replicate phenomena like the Bouba/Kiki effect, where abstract shapes are associated with specific sounds, extending this to emotional associations.

Key Results

The study found a positive correlation of approximately 0.40 between GPT-4's emotion ratings for nonsense words and human ratings. While this correlation is lower than the inter-annotator agreement among humans (around 0.69), it is significantly higher than the correlation achieved by a traditional regression model trained on real words (0.17). The highest correlation was observed for "disgust" (0.47), and the lowest for "surprise" (0.26). All reported correlations were statistically significant (p < 0.05).

Limitations and Future Work

The authors acknowledge that the correlation between LLM and human ratings (0.40) is not perfect, indicating that LLMs do not fully replicate human emotional associations. A key limitation is that the experiment was conducted with English speakers, and it remains an open question whether LLMs trained primarily on English data can generalize to other languages or cultural contexts. Future work proposed includes assigning more diverse personas to LLM prompts to explore variations in ratings based on factors like gender, age, and nationality, and further investigating the specific factors that influence an LLM's nonword interpretation.

Summary updated

BibTeX
@inproceedings{miyakawa2024do,
  author = {Yui Miyakawa and Chihaya Matsuhira and Hirotaka Kato and Takatsugu Hirayama and Takahiro Komamizu and Ichiro Ide},
  title = {{Do LLMs Agree with Humans on Emotional Associations to Nonsense Words?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure