TL;DR
This paper investigates how computational models of language learn grammatical "slot-constraints" (like word order or semantic roles) from usage data. They found that models prioritizing word associations, rather than just raw frequency, generalize better across eight languages, suggesting that how words co-occur is more crucial than how often they appear in isolation for learning grammar.
What Problem Does This Paper Address?
Construction Grammar (CxG) views language as a system of "constructions" – pairings of form and meaning. A key challenge in CxG, particularly for usage-based approaches, is understanding how the specific constraints within these constructions (e.g., "a noun must follow a verb") are learned from exposure to language. Existing computational CxG models often require manual specification of these constraints. This paper aims to bridge this gap by evaluating two competing computational models for learning these slot-constraints: one based on simple frequency of co-occurrence and another based on the statistical association between words. The core question is: which model better captures how humans learn grammatical patterns from language use?
Methods
The authors propose a computational pipeline for representing potential constructions and then evaluate two methods for selecting the "best" set of slot-constraints from this space:
- Frequency-Based Model: This approach prioritizes slot-constraints that appear frequently in the input data. It uses a fixed frequency threshold and then applies horizontal pruning to remove redundant, less specific constraints.
- Association-Based Model: This method uses the "bidirectional AP" (Association Probability) measure to quantify how strongly two slot-constraints are linked, considering both left-to-right and right-to-left relationships. It employs a search algorithm that follows these associations and uses a grid search to find the optimal association threshold for each language.
Both models are evaluated using the Minimum Description Length (MDL) metric, which quantifies how well a grammar compresses a test corpus. A lower MDL score (higher compression) indicates better generalization.
Datasets
The study uses eight languages: Arabic (ara), German (deu), English (eng), French (fra), Portuguese (por), Russian (rus), Spanish (spa), and Chinese (zho). For each language, a large background corpus (approximately 1 billion words) is used for:
- Training word embeddings.
- Determining the word-form lexicon.
- Calculating association measures.
- Learning and evaluating CxGs.
The corpora are primarily derived from web-crawled data, legislative texts, and newspaper articles. For evaluation, the data is divided into five independent test sets of 10 million words each per language.
Key Insights and Ideas
- Association Trumps Raw Frequency: The most striking insight is that association-based models consistently outperform frequency-based models in generalizing grammatical constraints. This suggests that the strength of the relationship between words is more indicative of grammatical structure than their sheer frequency.
- Context Matters: Association measures, by considering how often words co-occur relative to their individual frequencies, capture a more nuanced view of linguistic patterns than simple counts. This is crucial for understanding how less common but still grammatical constructions are learned.
- MDL as a Unified Evaluation Metric: The use of the MDL metric provides a principled, quantitative way to compare different grammar learning algorithms across multiple languages, allowing for a direct assessment of generalization ability.
- Cross-Linguistic Consistency: While the degree of difference varies, the superiority of association-based models holds across all eight languages tested, suggesting a fundamental principle of language acquisition.
Key Results
- The association-based model consistently achieved lower MDL scores (better compression/generalization) than the frequency-based model across all eight languages.
- The difference in performance was particularly pronounced for Russian, with a 40.21% compression gap.
- For French and Portuguese, the performance gap was smaller (2.76% and 6.22% respectively), indicating that frequency plays a more significant role in these languages compared to others, but association still provided better generalization.
- Analysis of the MDL components (grammar size L1, encoded constructions L2{C}, and unencoded data L2{R}) revealed that while association-based models generally had better L2{C} (more correctly encoded constructions), they sometimes had higher L2{R} (more unencoded data) in languages where the performance gap was smaller, suggesting a trade-off.
Limitations and Future Work
- Representation Pipeline: The authors acknowledge that the specific choices made in the representation pipeline (e.g., word embeddings, POS tagger) are not explored counter-factually. Future work could investigate the impact of these decisions.
- Non-Contiguous Constructions: The paper touches upon the challenge of handling non-contiguous constructions, which is a significant area for future research.
- Probability Estimation: The current model estimates probabilities for slot-fillers at the construction level. Future work could explore more nuanced probability estimation methods.
- Psycholinguistic Validation: While MDL provides a computational evaluation, the authors suggest that psycholinguistically-annotated datasets or qualitative distinctions could offer complementary insights into human cognition.
- Constituency: The paper briefly discusses how constituency might be handled within CxG, suggesting this as an avenue for further exploration.
Summary updated