This paper, presented at COLING-ACL 2006, tackles a fundamental problem in linguistics and Natural Language Processing (NLP): how do we move beyond a binary "correct vs. incorrect" view of grammar to a nuanced, mathematical scale of "grammaticality"?
TL;DR
The authors propose a method to quantify sentence grammaticality using Property Grammars, a constraint-based framework. By calculating a "Grammaticality Index" (GI) that weighs satisfied versus violated syntactic rules, they can accurately predict human acceptability judgments, even for "broken" or non-canonical sentences.
What Problem Does This Paper Address?
Traditional syntactic formalisms are often "all-or-nothing." If a sentence follows every rule in a grammar, it is grammatical; if it breaks even one, the parser often fails, and the sentence is labeled ungrammatical.
However, human language is messy. We encounter "ill-formed" sentences constantly—in spoken conversation, emails, or from non-native speakers—yet we still perceive degrees of "correctness." A sentence with a minor word-order error feels more "acceptable" than one that is a total word salad. The authors aim to bridge this gap by creating a robust, automated way to rank sentences by their degree of grammaticality, which is crucial for both improving NLP parsers and understanding human cognitive processing.
Methods
The researchers utilize Property Grammars (PG), a formalism where a grammar is not a set of generative rules, but a collection of constraints (properties) that categories (like Noun Phrases or Verb Phrases) should satisfy. These properties include:
- Linear Precedence: Does the determiner come before the noun?
- Uniqueness: Is there only one determiner?
- Obligation: Is the head of the phrase present?
- Exclusion: Can these two categories co-occur?
The Grammaticality Index (GI)
When a sentence is parsed, the system produces a "characterization"—a list of which properties were satisfied ($P^+$) and which were violated ($P^-$). To turn this into a single score, the authors developed a multi-layered calculation:
- Satisfaction Ratio (SR): The percentage of evaluated rules that were actually met.
- Completeness Coefficient (CC): How much of the total grammar was actually "activated" or relevant to the sentence.
- Quality Index (QI): A weighted score where different types of rules (e.g., word order vs. repetition) are given different "weights" based on their importance.
- Propagation: The final GI isn't just for the whole sentence; it’s an average that accounts for the quality of every embedded phrase (NP, VP, etc.). A high-quality sentence structure can "compensate" for a minor error deep inside a sub-phrase.
Datasets
The authors evaluated their model using:
- A custom French dataset: 60 experimental sentences designed with specific syntactic violations (Noun Phrase, Verb Phrase, Adjective Phrase, and Prepositional Phrase errors).
- Human Judgments: 44 native French speakers provided "acceptability" scores using Magnitude Estimation—a psycholinguistic technique where participants give numeric scores to sentences relative to a "reference" sentence.
Key Insights and Ideas
- Grammaticality is Gradient: The paper reinforces the idea that grammaticality isn't a binary state but a "gradience." By treating syntax as a system of soft constraints rather than hard rules, we can model language more realistically.
- Structural Compensation: One of the most interesting features of their GI formula is that it accounts for the "goodness" of embedded constituents. A sentence might have a local error, but if the rest of the hierarchy is solid, the overall grammaticality score remains relatively high.
- Weighting Matters: Not all grammatical rules are created equal. The authors found that violating "Linearity" (word order) or "Obligation" (missing a head word) is perceived as much more severe by humans than violating "Uniqueness" (repetition).
Key Results
The model's predictions showed a strong correlation with human intuition:
- Initial Correlation: The Grammaticality Index achieved a correlation coefficient of $\rho = 0.76$ with human acceptability scores.
- Refined Correlation: When the authors removed Prepositional Phrase (PP) violations—which humans judged inconsistently due to semantic and context-heavy factors—the correlation jumped to $\rho = 0.87$.
- Simplicity Wins: Surprisingly, the best fit was achieved using only two distinct weights for rules (weights of 2 and 5), suggesting that a highly complex weighting system might not be necessary to capture the essence of human judgment.
Limitations and Future Work
- The "Semantic" Gap: The model is purely syntactic. It cannot account for sentences that are grammatically perfect but semantically nonsensical (e.g., "Colorless green ideas sleep furiously").
- PP Instability: The authors noted that human judgments on Prepositional Phrases were "unstable," likely because the acceptability of a PP often depends on the specific meaning of the preposition, which the current model doesn't track.
- Scaling Up: The study was limited to 60 sentences. The authors acknowledge that the next step is validating this approach on large-scale, real-world corpora to see if the GI holds up in the wild.
Summary updated