TL;DR
This paper introduces the Syntactic Acceptability Dataset (SAD), a new resource of 1,000 English sentences with grammaticality and acceptability judgments. It aims to bridge the gap between linguistic theory and machine learning by providing a large-scale, publicly available dataset that can be used to investigate debates in linguistics and improve NLP models.
What Problem Does This Paper Address?
Linguistic theory often relies on "grammaticality judgments" made by experts to understand the rules of language. However, these judgments can be abstract and may not always align with how native speakers intuitively perceive sentence "goodness" (acceptability). This disconnect poses a challenge for both linguistic research and for building natural language processing (NLP) models that can truly understand and generate human-like language. Existing datasets are either too small, not publicly available, or focus solely on grammaticality, leaving a need for a comprehensive resource that captures both aspects of linguistic evaluation.
Methods
The authors created the Syntactic Acceptability Dataset (SAD) by sampling 500 sentences from the CoLA dataset (which itself is derived from syntax textbooks) and 500 sentences from the journal Linguistic Inquiry. Each sentence in SAD is accompanied by its original grammaticality judgment. Crucially, they then collected acceptability judgments for these sentences through a crowdsourcing platform (Prolific.com), ensuring participants were native English speakers and adhering to high experimental standards. They also measured response times and implemented quality control measures to ensure reliable data. For machine learning analysis, they fine-tuned Transformer models on their dataset under various conditions: predicting grammaticality, predicting acceptability, and predicting binary end-point acceptability.
Datasets
- Syntactic Acceptability Dataset (SAD): A newly introduced dataset comprising 1,000 English sentences. It includes:
- Grammaticality judgments (from original sources).
- Acceptability judgments (collected via crowdsourcing on a 7-point scale, then normalized and binarized).
- Source information (textbook or journal).
- CoLA (Corpus of Linguistic Acceptability): Used as a source for 500 sentences in the SAD.
- Linguistic Inquiry: A journal from which 500 sentences were sampled.
Key Insights and Ideas
- Grammaticality vs. Acceptability Divergence: The dataset highlights that while grammaticality and acceptability often align (around 83% convergence), there's a significant minority of cases where they diverge. This suggests that linguistic formalisms (grammaticality) and native speaker intuition (acceptability) are distinct concepts, with laypeople sometimes finding sentences acceptable that linguists deem ungrammatical.
- Machine Learning Performance Gap: NLP models, particularly Transformers, struggle significantly with predicting grammaticality. However, they perform considerably better when predicting acceptability, and even better when predicting binary end-point acceptability. This is a novel finding suggesting that acceptability might be a more learnable linguistic phenomenon for current models than strict grammaticality.
- Gradience is Pervasive: Acceptability is not a binary concept but a gradient one. The data shows a near-linear increase in acceptability as items are ranked, with a substantial portion of items falling into "in-between" categories, reinforcing the idea that language use exists on a spectrum.
Key Results
- Data Convergence: Approximately 83.3% of items in SAD show convergence between their grammaticality and binarized acceptability labels. This rate is higher for textbook-sourced items (85.8%) than journal-sourced items (80.8%).
- Machine Learning Performance:
- Grammaticality prediction accuracy: 70%
- Acceptability prediction accuracy: 83%
- End-point acceptability prediction accuracy: 94%
- A baseline model (predicting based on word order) achieved 97% accuracy, suggesting that simple word order is not the sole determinant of acceptability.
- Acceptability Distribution: 28.3% of items fall into the middle third of the acceptability scale, indicating a significant degree of gradience.
Limitations and Future Work
- Dataset Size: While the largest of its kind, the current 1,000-item dataset is still relatively small for extensive machine learning training. The authors plan to scale it up to approximately 15,000 items by incorporating more data from existing resources.
- Further Annotations: Future work will involve adding expert annotations for syntactic constructions and potentially prosodic information and dependency parses to enrich the dataset's analytical capabilities.
- Deeper Analysis: The authors intend to conduct more in-depth analyses, including examining response times for correlations with unacceptability and exploring the "micro-factors" that contribute to observed "macro-trends" in acceptability judgments.
- Model Robustness: The current machine learning findings are based on a limited dataset, and further validation with a larger dataset is necessary to solidify the observed performance differences between grammaticality and acceptability prediction.
Summary updated