RNN Simulations of Grammaticality Judgments on Long-distance Dependencies

Shammur Absar Chowdhury, Roberto ZamparelliCOLING 2018

TL;DR

This paper investigates whether Recurrent Neural Networks (RNNs), specifically LSTMs and GRUs, trained on a language modeling task can learn to identify ungrammatical sentences caused by long-distance dependency violations. The findings suggest that while these models can capture some linguistic phenomena, their performance is heavily influenced by processing factors, raising questions about their ability to learn abstract grammaticality.

What Problem Does This Paper Address?

For decades, theoretical linguistics has relied on native speaker intuitions about grammaticality to understand the human language faculty. A key debate is whether this linguistic competence is innate or learned entirely from exposure. This paper explores the latter by asking if artificial neural networks, trained solely on large text corpora (mimicking exposure), can replicate human-like grammaticality judgments, particularly for complex structures like long-distance dependencies. If NNs can learn these judgments without explicit linguistic rules, it supports the idea that language can be learned from input alone. Conversely, if they fail on certain phenomena, it might suggest those phenomena rely on innate, specialized linguistic mechanisms.

Methods

The researchers employed two types of RNN architectures: Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). These models were trained on a standard language modeling task: predicting the next word in a sequence. This approach is chosen because it's considered a more "unbiased" learning task compared to directly training a classifier for grammaticality judgments. After training, the models were evaluated on their ability to distinguish between grammatical and ungrammatical sentences, focusing on specific types of linguistic violations.

Two primary evaluation measures were used:

  • Global Measures: Perplexity (PPL) and average cross-entropy loss (ACEL). These measures assess how "confused" the model is by a sentence, with lower values indicating better prediction and thus, potentially, better understanding of grammaticality.
  • Local Measures: Normalized log probability of sentence-ending punctuation (P(FS/QM)). This tracks the model's expectation of sentence completion.

Datasets

The models were trained on a large English corpus derived from Wikipedia and a subset of UKWaC. This corpus comprised approximately 136 million words with a vocabulary of about 0.1 million unique tokens. The text was preprocessed to remove URLs, email addresses, emoticons, and bracketed text, and rare words were replaced with an <unk> token. Numbers were also standardized.

For evaluation, the researchers created specific test sets of artificially constructed sentences designed to probe different types of long-distance dependency violations. These sets were generated from schemata to ensure systematic variation and comparison.

Key Insights and Ideas

  • Processing vs. Grammaticality: The study highlights a significant challenge: RNNs struggle to disentangle pure grammatical violations from processing difficulties. Factors like sentence length and embedding depth heavily influence the model's performance, suggesting that what appears as a grammaticality judgment might, in fact, be a reflection of processing load.
  • Sensitivity to Structure: Despite the processing influence, the models did show sensitivity to certain structural properties, like the preference for subject over object relative clauses, mirroring human findings.
  • Island Violations Remain Tricky: The most complex syntactic structures, known as "syntactic islands," proved particularly challenging for the RNNs. While the models showed some ability to distinguish grammatical from ungrammatical island cases, the performance was often marginal, and the interpretation of the results was complicated by other factors.
  • The "Abstract Grammaticality" Question: The paper casts doubt on whether current RNNs, trained on language modeling, can induce an abstract notion of grammaticality. Their performance seems more tied to statistical patterns and processing ease rather than a deep, rule-based understanding of syntax.

Key Results

  • Task A (Subject vs. Object Relatives): The RNN models performed better on subject relatives than object relatives, aligning with human psycholinguistic findings. They also showed a preference for "that" over "who" as a relative pronoun and for null relative pronouns in object clauses, consistent with corpus frequencies.
  • Task B (Wh-Extractions and Full Interpretation Violations): The models achieved high accuracy (91.45%) in classifying Wh-extraction cases, correctly identifying the ungrammaticality of extra arguments. They also showed a preference for personal pronouns as gap-fillers over indefinite or demonstrative NPs.
  • Task C (Subject and Relative Island Violations): The results for island violations were more mixed. While the models showed some ability to distinguish violations, the performance was sensitive to factors like sentence type (Wh-interrogative vs. Y/N vs. affirmative) and the position of the violation (subject vs. object). The RS case (relative clause + subject island) yielded the highest perplexity, suggesting it was the most difficult.

Limitations and Future Work

The authors acknowledge that their models' performance is highly sensitive to processing factors, making it difficult to isolate pure grammaticality judgments. They also note that the training data's style (mostly encyclopedic) might not perfectly represent all registers of English.

Future work suggested includes:

  • Exploring alternative evaluation measures that better disentangle processing and grammaticality.
  • Using bidirectional LSTMs to potentially improve performance.
  • Developing methods for providing minimal pairs to NNs, similar to how human judgments are elicited, to better understand how they learn from direct comparisons.
  • Investigating whether NNs can learn innate-grammar-free language learners and what abstract properties they can derive from input.

Summary updated

BibTeX
@inproceedings{chowdhury2018rnn,
  author = {Shammur Absar Chowdhury and Roberto Zamparelli},
  title = {{RNN Simulations of Grammaticality Judgments on Long-distance Dependencies}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure