Language Models Largely Exhibit Human-like Constituent Ordering Preferences

Ada Defne Tur, Gaurav Kamath, Siva Reddy

Abstract

Though English sentences are typically inflexible vis-à-vis word order, constituents often show far more variability in ordering. One prominent theory presents the notion that constituent ordering is directly correlated with constituent weight: a measure of the constituent's length or complexity. Such theories are interesting in the context of natural language processing (NLP), because while recent advances in NLP have led to significant gains in the performance of large language models (LLMs), much remains unclear about how these models process language, and how this compares to human language processing. In particular, the question remains whether LLMs display the same patterns with constituent movement, and may provide insights into existing theories on when and how the shift occurs in human language. We compare a variety of LLMs with diverse properties to evaluate broad LLM performance on four types of constituent movement: heavy NP shift, particle movement, dative alternation, and multiple PPs. Despite performing unexpectedly around particle movement, LLMs generally align with human preferences around constituent ordering.

Introduction: The Principle of End-Weight

In English, while the basic subject-verb-object order is relatively strict, the arrangement of phrases following the verb is surprisingly flexible. A central observation in linguistics is the "principle of weight" or "end-weight"—the tendency to place longer, more complex phrases at the end of a sentence. For example, while a speaker might say "I met the man at the park," they are far more likely to shift the direct object if it is complex: "I met at the park the tall man selling water to marathon runners."

As Large Language Models (LLMs) become the primary tools for processing and generating human language, it is vital to understand whether they "understand" these subtle ordering preferences or merely calculate statistical noise. This study investigates whether LLMs exhibit human-like sensitivities to constituent weight and identifies which specific measures of "weight" best explain their behavior.

Experimental Framework

To map word order preferences, the researchers evaluated a diverse suite of open-source models, including the GPT-2 family, Llama-3, Mistral, and OLMo. The methodology employed a three-pronged approach to quantify model behavior:

  1. Preference Scoring ($M_{preference}$): Models were tasked with assigning log probabilities to "normal" (unshifted) sentences versus their "shifted" counterparts. A higher score for the shifted version indicates a model preference for that specific word order.
  2. Regression Analysis (GAMMs): Using Generalized Additive Mixed Models, the authors ablated different measures of weight—word length, syllable count, token count, and modifier count—to determine which metric most accurately predicted model behavior.
  3. Human Correlation: To establish a ground truth, a crowdsourced study with 126 native English speakers provided "naturalness" ratings for 500 sentence pairs. These human scores were then compared to model preferences using Spearman correlation.
Overview of the experimental pipeline: from constituent movement data to human-model preference correlation.

Data and Evaluation

The study utilized a combination of synthetic and mined data to ensure both breadth and realism. A novel dataset of approximately 400,000 minimal pairs was generated via templates covering four linguistic phenomena: Heavy NP Shift (HNPS), Particle Movement (PM), Dative Alternation (DA), and Multiple PP Shift (MPP). These were supplemented by 700 naturally occurring sentences extracted from the Penn Treebank-2 to ground the synthetic results in real-world usage.

Key Findings

The Syllable Surprise

One of the most unexpected findings was that syllable weight served as the strongest predictor of ordering preferences across most models. This is surprising because LLMs process text as discrete tokens rather than phonological units. This suggests that models either implicitly learn phonological-like complexity or that syllables serve as a superior proxy for the "information density" that triggers a linguistic shift.

Model preference scores for Heavy NP Shift as a function of different weight ratios, highlighting the strong predictive power of syllable weight.

The Instruction-Tuning Paradox

The research revealed a striking "instruction-tuning paradox": models that underwent Reinforcement Learning from Human Feedback (RLHF), such as Llama-3-Instruct, consistently showed lower correlation with human linguistic intuitions than their base model counterparts. This suggests that the instruction-tuning process might inadvertently flatten or "break" some of the nuanced linguistic structures acquired during the initial pre-training phase.

General Alignment and Anomalies

Overall, LLMs follow the end-weight principle. As the relative weight of a constituent increases, the models' preference for shifting that constituent to the end of the sentence increases, eventually hitting a "ceiling effect" similar to human processing. Spearman correlations were highest for Dative Alternation (up to 0.65) and Heavy NP Shift (up to 0.51). However, models behaved erratically with Particle Movement (e.g., "She looked up the answer" vs. "She looked the answer up"), showing a sharp drop in preference for shifting as weight increased initially—a direct contradiction of standard linguistic theory.

Conclusion

While LLMs generally mirror human constituent ordering preferences, their behavior is not a perfect reflection of human intuition. The superior performance of smaller, older base models over newer instruction-tuned variants suggests that current alignment techniques may be sacrificing linguistic nuance for task performance. Future work should expand these investigations to non-English languages to determine if these patterns are universal or specific to English syntax.

Summary updated

Abstract

Though English sentences are typically inflexible vis-\`a-vis word order, constituents often show far more variability in ordering. One prominent theory presents the notion that constituent ordering is directly correlated with constituent weight: a measure of the constituent's length or complexity. Such theories are interesting in the context of natural language processing (NLP), because while recent advances in NLP have led to significant gains in the performance of large language models (LLMs), much remains unclear about how these models process language, and how this compares to human language processing. In particular, the question remains whether LLMs display the same patterns with constituent movement, and may provide insights into existing theories on when and how the shift occurs in human language. We compare a variety of LLMs with diverse properties to evaluate broad LLM performance on four types of constituent movement: heavy NP shift, particle movement, dative alternation, and multiple PPs. Despite performing unexpectedly around particle movement, LLMs generally align with human preferences around constituent ordering.
BibTeX
@inproceedings{tur2025language,
  author = {Ada Defne Tur and Gaurav Kamath and Siva Reddy},
  title = {{Language Models Largely Exhibit Human-like Constituent Ordering Preferences}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.126},
}
Figure