Abstract The English Preposing in PP construction (PiPP; e.g., Happy though / as we were) is extremely rare but displays an intricate set of stable syntactic properties. How do people become proficient with this construction despite such limited evidence? It is tempting to posit innate learning mechanisms, but present-day large language models seem to learn to represent PiPPs as well, even though such models employ only very general learning mechanisms and experience very few instances of the construction during training. This suggests an alternative hypothesis on which knowledge of more frequent constructions helps shape knowledge of PiPPs. I seek to make this idea precise using model-theoretic syntax (MTS). In MTS, a grammar is essentially a set of constraints on forms. In this context, PiPPs can be seen as arising from a mix of construction-specific and general-purpose constraints, all of which seem inferable from general linguistic experience.
Introduction
In the study of linguistics, the "Poverty of the Stimulus" argument posits that children acquire complex grammatical rules despite receiving insufficient data from their environment. This has traditionally led to the conclusion that certain aspects of grammar must be innate. Christopher Potts investigates this claim through the lens of a specific "syntactic nut": the English Preposing in PP (PiPP) construction. Examples such as "Brilliant linguists though they were..." or "Happy as they seemed..." represent a rare but highly structured linguistic pattern.
PiPP is characterized by several unique constraints: it is vanishingly rare in natural speech, it is restricted to specific prepositions (though and as), and it involves a long-distance dependency where a phrase is moved to the front, leaving a "gap" in its original position.
The Syntactic Structure of PiPP
To understand how PiPP functions, we can model it using Model-Theoretic Syntax (MTS), which views grammar as a collection of constraints. A typical PiPP structure, such as "incredible though it seems", can be decomposed into a hierarchical tree.
As shown in the figure above, the construction consists of a Prenucleus (the preposed adjective phrase "incredible") and a Nucleus. The Nucleus is a Prepositional Phrase (PP) where the Head is a specific preposition like though. The Complement of this preposition is a clause ("it seems") that contains a GAP. This gap is co-indexed with the Prenucleus, indicating that "incredible" is the semantic object of "seems".
Quantifying Rarity and Complexity
Using massive datasets like the BookCorpusOpen and the C4 web-crawl, Potts quantified the extreme scarcity of PiPP. It appears in fewer than 0.03% of sentences. For comparison, common structures like relative clauses are hundreds of times more frequent.
Despite this rarity, the construction exhibits "unboundedness." Potts identified natural examples where the dependency spans multiple clauses, such as: "Well-intentioned though many people may have imagined that the CIA probably thought they were...". This confirms that PiPP follows the same complex rules as more common long-distance dependencies (like wh-questions).
LLMs as Syntactic Probes
The central question is whether general-purpose learning mechanisms can master PiPP without innate linguistic modules. Potts tested Large Language Models (LLMs) like Pythia and BERT to see if they "expect" these structures.
- Gap Licensing: Using Pythia 12B, Potts measured "surprisal." The model correctly predicted the presence of a gap only when a corresponding "filler" (the preposed phrase) was present at the start of the sentence.
- Head Restrictions: BERT demonstrated a categorical preference for as and though in these contexts, finding similar words like although highly surprising, despite although being more common in general usage.
Conclusion
The success of LLMs in representing PiPP constructions suggests that "syntactic nuts" may not require hard-wired innate knowledge. Instead, these rare structures likely emerge from a combination of general-purpose constraints and generalizations from more frequent, related patterns. While the "data gap" between AI training and human childhood remains a significant factor, these findings provide a computational proof-of-concept for how complex, rare grammar can be learned from experience.
Summary updated