What to Predict? Exploring How Sentence Structure Influences Contrast Predictions in Humans and Large Language Models

Shuqi Wang, Xufeng Duan, Zhenguang CaiNAACL 2025

TL;DR

This study investigates how sentence structure influences predictions about contrasting elements in Mandarin Chinese, comparing human behavior to that of large language models (LLMs). The findings reveal that while both humans and LLMs are sensitive to sentence structure, LLMs exhibit a stronger structural effect, with GPT-4o showing the closest alignment to human spoken data.

What Problem Does This Paper Address?

Predictive processing is a cornerstone of how both humans and LLMs understand and generate language. When we encounter language, we're constantly anticipating what comes next. This anticipation is crucial for efficient comprehension and is a core mechanism in LLMs, which are designed to predict the next token in a sequence.

A key aspect of this predictive process is understanding contrast. When something is presented with a contrastive marker (like "but" or "not"), we expect an alternative to a previously mentioned element. However, in complex sentences, it's not always clear which element is being contrasted. For example, in "She gave the girl the candy, not...", the contrast could be about the recipient ("the girl"), the theme ("the candy"), or even the verb ("gave"). This ambiguity is particularly interesting in ditransitive constructions (sentences with two objects, like "give X to Y"), which allow for flexible word order and thus can shift focus.

This paper tackles the question of how sentence structure, specifically in Mandarin ditransitive constructions, influences these contrast predictions in both humans and LLMs. It also explores whether the modality of language (written vs. spoken) plays a role in this influence.

Methods

The researchers employed a sentence continuation task. Participants (both human and LLM) were presented with the first part of a sentence, ending with a contrast marker (e.g., "She gave the girl the candy, but not..."), and asked to provide the most natural completion.

For human participants, three groups were tested:

  • Written stimuli: Participants read the sentences.
  • Original spoken stimuli: Participants heard naturally spoken sentences.
  • Normalized spoken stimuli: Participants heard spoken sentences where prosodic features were manipulated to control for variations in stress and duration, allowing for a clearer isolation of structural effects.

Three state-of-the-art LLMs were used: OpenAI's GPT-4o, Meta's LLaMA-3, and Alibaba's Qwen-2.5. These models were chosen for their performance, diversity (closed-source vs. open-weight, English-dominant vs. Chinese-dominant), and were prompted to complete the same sentence fragments as the human participants.

The continuations were then categorized into three types of contrasts:

  1. Theme Contrast (T): Contrasting the object being transferred (e.g., "not the cake").
  2. Recipient Contrast (R): Contrasting the person receiving the object (e.g., "not the boy").
  3. Verb or Verb Phrase Contrast (VP): Contrasting the action itself (e.g., "not bought").

Statistical analyses, including t-tests and linear mixed-effects models, were used to compare the frequencies of these contrast types across different conditions (DO vs. PO, written vs. spoken, human vs. LLM). Pearson correlations were calculated to assess the similarity between LLM and human prediction patterns.

Datasets

The study used a custom dataset of 42 experimental items, each featuring a Mandarin ditransitive construction followed by a contrast marker. These items were based on previous research and included 14 different ditransitive verbs. The items were presented in two structural conditions: Double Object (DO) and Prepositional Object (PO). Additionally, 90 filler sentences were included to ensure participants engaged with a variety of linguistic structures. No new benchmarks were introduced; the focus was on analyzing existing linguistic phenomena with both human and model participants.

Key Insights and Ideas

  • Theme is King (Mostly): For humans, the theme (what is being given) was the most frequent element to be contrasted, regardless of sentence structure. This supports the "Thematic Hierarchy Hypothesis," suggesting themes are often the default focus.
  • Structure Matters, Especially in Speech: While theme was dominant, sentence structure did modulate predictions. In spoken language, the position of elements became more influential, with sentence-final elements (like the recipient in PO constructions) drawing more attention for contrast.
  • LLMs are Structure-Sensitive Superstars: LLMs, particularly GPT-4o, showed a much stronger effect of sentence structure on contrast predictions than humans. They seemed to over-index on linear order, mimicking spoken language's sensitivity to final elements.
  • Spoken Language is the LLM's Native Tongue (Figuratively): LLMs' predictions aligned more closely with spoken human data than written data. This suggests that the incremental, sequential nature of speech processing might be more akin to how LLMs process information.

Key Results

  • Human Participants: Theme contrasts were the most frequent across all modalities (written, spoken). However, sentence structure had a significant effect, with spoken language showing a stronger influence of word order on contrast predictions compared to written language.
  • LLMs: All three LLMs also favored theme contrasts. Crucially, the structural effect (how DO vs. PO influenced predictions) was significantly larger in all LLMs compared to humans. GPT-4o showed the highest correlation with human data, especially spoken data.
  • LLM vs. Human Alignment: LLMs' predictions correlated more strongly with spoken human data than written human data, indicating a better alignment with the sequential processing of speech.

Limitations and Future Work

The authors acknowledge that the sentence continuation task is an "offline" method that blends comprehension and production. Future research could benefit from more online, time-sensitive methods like eye-tracking to better understand the moment-by-moment processing of focus and contrast.

Additionally, while three prominent LLMs were studied, the rapid advancement of AI means that future work should include newer models and explore the impact of different training data (e.g., models trained primarily on Chinese vs. English). Investigating the internal mechanisms of LLMs, such as attention weights, could also shed light on why they exhibit these specific prediction patterns and how they differ from human cognitive processes.

Summary updated

BibTeX
@inproceedings{wang2025what,
  author = {Shuqi Wang and Xufeng Duan and Zhenguang Cai},
  title = {{What to Predict? Exploring How Sentence Structure Influences Contrast Predictions in Humans and Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure