Do large language models resemble humans in language use?

Zhenguang Cai, Xufeng Duan, David Haslett, Shuqi Wang, Martin PickeringACL 2024

TL;DR

Large Language Models (LLMs) like ChatGPT and Vicuna are surprisingly human-like in their language processing, replicating human patterns in 10 out of 12 psycholinguistic experiments. This suggests LLMs might be valuable tools for cognitive scientists studying human language.

What Problem Does This Paper Address?

The rapid advancement of LLMs has sparked a debate: do these models merely mimic human language through statistical pattern matching, or do they genuinely reflect aspects of human language processing? While LLMs excel at grammar, it's unclear if they also exhibit human-like behaviors in areas like sound-shape associations, semantic priming, and inferencing. This paper aims to bridge this gap by systematically comparing LLM performance to human performance across a range of psycholinguistic experiments.

Methods

The researchers designed 12 pre-registered psycholinguistic experiments to test ChatGPT and Vicuna. These experiments covered various linguistic levels, from phonology (sound-shape, sound-gender associations) to semantics (word length, word meaning priming, implausible sentence interpretation, semantic illusions) and syntax (structural priming, syntactic ambiguity resolution), and discourse (implicit causality, drawing inferences). For each experiment, LLM responses were analyzed using mixed-effects modeling, and results were compared to established human data. The LLMs were prompted using a simulated human interlocutor, and their responses were analyzed for specific linguistic phenomena.

Datasets

The paper does not introduce new datasets for training LLMs. Instead, it utilizes existing experimental paradigms and materials from prior human studies to create prompts for the LLMs. For example, stimuli for sound-shape association were adapted from Sidhu & Pexman (2017), and for word meaning priming, from Rodd et al. (2013). The paper also references preregistered data and analytical codes available on OSF.

Key Insights and Ideas

  • LLMs are more than just pattern matchers: The study provides strong evidence that LLMs exhibit human-like linguistic behaviors beyond simple statistical correlations, suggesting emergent cognitive processes.
  • Context is king (for some LLMs): ChatGPT, in particular, demonstrated an ability to use context to interpret implausible sentences and resolve ambiguities, mirroring human flexibility.
  • Model size matters: ChatGPT consistently outperformed Vicuna, suggesting that larger models may better capture the nuances of human language use.
  • Transformer architecture's role: The authors hypothesize that the transformer architecture's ability to weigh contextual information might explain many of the observed human-like behaviors.

Key Results

  • High Replication Rate: ChatGPT replicated human patterns in 10 out of 12 experiments, while Vicuna replicated them in 7 out of 12.
  • Sound-Shape Association: Both LLMs correctly associated round-sounding words with round shapes and spiky-sounding words with spiky shapes.
  • Semantic Priming: Both models showed susceptibility to word-meaning priming, accessing recently encountered meanings of ambiguous words.
  • Structural Priming: Both LLMs exhibited structural priming, repeating syntactic structures they had recently encountered.
  • Implausible Sentence Interpretation: ChatGPT, but not Vicuna, nonliterally interpreted implausible sentences, similar to humans.
  • Semantic Illusions: ChatGPT, unlike Vicuna, showed a human-like tendency to overlook semantic illusions when the erroneous word was semantically similar to the intended one.
  • Interlocutor Sensitivity: Both LLMs adjusted their word meaning access and lexical retrieval based on the interlocutor's dialect, mirroring human behavior.
  • Exceptions: Neither model consistently preferred shorter words for less informative content, nor did they use context to resolve syntactic ambiguities as effectively as humans.

Limitations and Future Work

The authors acknowledge several limitations:

  • Task Selection Bias: The choice of 12 tasks might have inadvertently favored LLM strengths.
  • Experimental Design Discrepancies: Differences in materials and procedures between LLM and human studies could influence results.
  • Lack of Direct Human Comparison: Many experiments couldn't directly compare LLM and human behavior due to data unavailability.

Future work could involve using LLMs as cognitive models to generate hypotheses, estimate effect sizes, and model language development. The authors suggest that the observed convergences and divergences likely stem from the transformer architecture, and further research could explore this relationship.

Summary updated

BibTeX
@inproceedings{cai2024do,
  author = {Zhenguang Cai and Xufeng Duan and David Haslett and Shuqi Wang and Martin Pickering},
  title = {{Do large language models resemble humans in language use?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure