Abstract
This study investigates the extent to which Large Language Models (LLMs) align with human "constituent ordering" preferences, specifically the "Principle of Weight" where longer, more complex phrases are shifted toward the end of a sentence. By evaluating model probabilities against human judgments across diverse linguistic phenomena, we find that while LLMs generally mirror human intuition, instruction-tuning often degrades this alignment. Furthermore, syllable weight emerges as a surprisingly strong predictor of model behavior, suggesting that LLMs induce high-level linguistic structures beyond simple token frequencies.
Introduction
In English, while basic word order is relatively fixed, the post-verbal domain allows for significant flexibility. Humans typically follow the "Principle of Weight," preferring to place "heavy" (long or complex) constituents at the end of a sentence to facilitate processing. For instance, "I met the man at the park" is standard, but "I met at the park the tall man selling water to marathon runners" is preferred when the noun phrase is sufficiently complex. This research explores whether LLMs possess a similar structural "understanding" or if their outputs are merely a byproduct of statistical token frequencies.
Methods
The researchers utilized a surprisal-based approach to calculate a Preference Score ($M_{preference}$), defined as the difference in log-probabilities assigned by a model to a "shifted" sentence versus its "unshifted" counterpart. Alignment is defined as the model assigning higher probability to the version preferred by humans.
Datasets
The experimental framework relied on three primary data sources:
- Synthetic Data: A novel dataset of ~400,000 minimal pairs testing Heavy NP Shift (HNPS), Particle Movement (PM), Dative Alternation (DA), and Multiple PP Shift (MPP).
- Mined Data: ~700 naturally occurring sentences from the Penn Treebank-2 to validate findings in real-world contexts.
- Human Judgments: A ground-truth baseline established via naturalness ratings from 126 native English speakers on a 7-point scale.
Statistical analysis was performed using Generalized Additive Mixed Models (GAMMs) to determine which weight measures—word length, syllable count, token count, or modifier weight—best explained model preferences.
Results
The Syllable Surprise and Weight Convergence
A key finding was that syllable weight often outperformed token weight as a predictor of LLM behavior. This is notable because LLMs operate on sub-word tokens rather than phonetic syllables, suggesting they induce abstract linguistic features during training. Additionally, as constituent weight increases, model preferences for shifting converge, mirroring the "ceiling effects" observed in human psycholinguistics.
The Instruction-Tuning Paradox
Contrary to the expectation that instruction-tuning (e.g., RLHF) makes models more "human-like," the study found that base models (such as Llama-3 or Mistral) often aligned more closely with human ordering preferences than their "Instruct" counterparts. This suggests that the tuning process may prioritize task-specific patterns at the expense of nuanced linguistic intuitions.
Performance Across Phenomena
While models demonstrated high correlation with human judgments in areas like Dative Alternation (Spearman correlations up to 0.65), they struggled significantly with Particle Movement (e.g., "She looked up the question" vs. "She looked the question up"). This indicates that certain grammatical movements remain challenging for current architectures to master.
Discussion and Limitations
The study highlights that while LLMs exhibit human-like syntactic preferences, these preferences are sensitive to the training regime. Limitations include the focus on English, which has specific ordering rules that do not generalize to all languages. Furthermore, the study evaluates model rating (via log-probabilities) rather than generation. Future work should investigate the stylistic impact of instruction-tuning datasets and expand the analysis to cross-linguistic contexts where "heavy" constituents may appear earlier in the sentence.
Summary updated