Abstract
Language models (LMs) tend to show human-like preferences on a number of syntactic phenomena, but the extent to which these are attributable to direct exposure to the phenomena or more general properties of language is unclear. We explore this with the English dative alternation (DO: "gave Y the X" vs. PO: "gave the X to Y"), using a controlled rearing paradigm wherein we iteratively train small LMs on systematically manipulated input. We focus on two properties that affect the choice of alternant: length and animacy. Both properties are directly present in datives but also reflect more global tendencies for shorter elements to precede longer ones and animates to precede inanimates. First, by manipulating and ablating datives for these biases in the input, we show that direct evidence of length and animacy matters, but easy-first preferences persist even without such evidence. Then, using LMs trained on systematically perturbed datasets to manipulate global length effects (re-linearizing sentences globally while preserving dependency structure), we find that dative preferences can emerge from indirect evidence. We conclude that LMs' emergent syntactic preferences come from a mix of direct and indirect sources.
Introduction
In English, the "dative alternation" allows speakers to express the same event in two ways: the Double Object (DO) form (She gave the boy the book) or the Prepositional Object (PO) form (She gave the book to the boy). Human choice between these forms is not random; it is governed by strong preferences for "short-first" (placing the shorter phrase first) and "animate-first" (placing animate recipients before inanimate themes).
While modern LMs exhibit these same preferences, the source of this knowledge is debated. Do models simply memorize specific dative examples (direct evidence), or do they learn global linguistic principles—like a general "short-first" rule—and apply them to datives (indirect evidence)? This study employs a "controlled rearing" approach to disentangle these sources.
Methods: Controlled Rearing
The researchers used a controlled rearing paradigm, training OPT-125M models from scratch on systematically manipulated versions of an 88-million-word subset of the 2023 BabyLM corpus.
- Direct Evidence Manipulation: Training data was modified so datives were either perfectly balanced, "swapped" (violating standard biases), or removed entirely (
no-datives). - Indirect Evidence Manipulation: The entire training corpus was "re-linearized" using dependency parses. Sentences were reordered globally based on length (e.g., a "short-first" version vs. a "long-first" version) while preserving grammatical structure.
- Evaluation: Models were evaluated on the DAIS dataset (5,000 human-scored dative pairs) and a custom test set of 2,000 pairs labeled for animacy and length.
Key Results
The study reveals that both direct and indirect evidence are vital, though they function differently across properties.
The Persistence of "Short-First"
Even in the no-datives model, where dative constructions were entirely absent from training, a significant "short-first" preference emerged ($r = -0.24$ correlation between length difference and preference). This demonstrates that models can generalize global structural patterns to specific syntactic alternations they have never seen.
Direct Evidence as an Anchor
While indirect evidence provides a baseline, direct evidence can override it. In the "swapped-dative" condition, where every dative example intentionally violated human-like biases, the standard preference was neutralized. Furthermore, the "default" model achieved a correlation of $r = 0.50$ with human judgments, nearly matching the performance of the much larger GPT-2 ($r = 0.59$).
Animacy vs. Length
The preference for animate recipients was harder to learn without direct examples. Unlike length, which appears to be a structural principle, animacy preferences seem more closely tied to specific lexical items (verbs like give or sell).
Conclusion
Dative preferences in LMs are a sub-case of broader linguistic principles of efficiency. By training models on a globally "long-first" corpus, researchers successfully flipped the dative preference, confirming that global ordering dictates local syntax. Future work should investigate if massive LLMs rely more on these emergent principles or on the sheer scale of memorized direct evidence.
Summary updated