TL;DR
This paper investigates whether Long Short-Term Memory (LSTM) networks, trained on raw text, can learn a fundamental syntactic constraint on language called Principle C. The study found that even with extensive training, LSTMs failed to exhibit sensitivity to Principle C, suggesting that current architectures may require additional inductive biases to fully grasp complex linguistic structures.
What Problem Does This Paper Address?
The paper delves into the ongoing debate about whether sophisticated language processing capabilities, like understanding coreference (which entity a pronoun refers to), can emerge solely from statistical learning on vast amounts of text, or if innate linguistic knowledge is required. Specifically, it examines Principle C, a syntactic rule stating that a pronoun cannot "c-command" its coreferent (a pronoun and a noun phrase cannot refer to the same entity if the pronoun is in a position that syntactically dominates the noun phrase). This constraint is crucial for understanding who is doing what to whom in a sentence. Previous research has shown that even young children respect Principle C, but it's unclear if purely data-driven models like LSTMs can learn it.
Methods
The researchers utilized a pre-trained LSTM language model, similar to those used in state-of-the-art Natural Language Processing (NLP) systems. They tested this model on psycholinguistic materials derived from two published experiments on coreference. The core of their methodology involved analyzing the LSTM's predicted probabilities for upcoming words, specifically focusing on how gender agreement between a pronoun and a subsequent noun phrase influenced these probabilities. They looked for an "interaction" between gender mismatch and the c-command relationship, expecting that a mismatch would be more surprising (and thus lead to lower probabilities) when coreference was syntactically licit (i.e., not violating Principle C).
To further probe the LSTM's capabilities, they conducted a second set of experiments where they artificially manipulated the training data to make the signal for Principle C more statistically salient. This involved modifying sentences to enforce gender matches when coreference was licit, thereby creating a stronger statistical trace of the constraint.
Datasets
The study used materials from two published psycholinguistic experiments on coreference (Kazanina et al., 2007). The LSTM was pre-trained on a large corpus of approximately 3 million English sentences (Gulordava et al., 2018). For name analysis, they used lists of the 100 most frequent male and female names from birth registers in England and Wales.
Key Insights and Ideas
- LSTMs struggle with abstract syntactic constraints: Despite being trained on massive datasets, LSTMs appear to have difficulty learning abstract syntactic rules like Principle C, which are not directly tied to surface-level word co-occurrence.
- Statistical salience matters, but isn't enough: Even when the statistical signal for Principle C was artificially amplified in the training data, the LSTM still failed to learn the constraint effectively. This suggests the issue might be deeper than just the clarity of the signal.
- Coreference is more than just word prediction: While LSTMs excel at predicting upcoming words and capturing some syntactic phenomena (like number agreement), understanding coreference, which relies on deeper structural understanding, remains a challenge.
- Human language acquisition is different: The stark contrast between the LSTM's performance and human children's ability to grasp Principle C highlights potential differences in learning mechanisms.
Key Results
- In Experiments 1 & 2, the LSTM's behavior was not comparable to human subjects. The model showed a general sensitivity to gender mismatch but did not consistently adjust its predictions based on whether Principle C was violated or not.
- The interaction term between gender mismatch and the c-command feature, which would indicate sensitivity to Principle C, was either in the "wrong sign" or only weakly significant in the initial experiments.
- Even after artificially enhancing the statistical signal for Principle C in the training data (Experiments 3 & 4), the interaction term remained non-significant, indicating that the LSTM could not leverage this enhanced signal to learn the constraint.
Limitations and Future Work
The authors acknowledge that their study focuses on a specific type of LSTM and training regime. They suggest that future work should investigate what additional enhancements or architectural changes are needed to enable neural networks to effectively handle coreference structures. This could involve exploring different network architectures, incorporating explicit linguistic knowledge, or developing novel training methodologies that better capture abstract syntactic rules. The possibility that the relevant signal for Principle C is inherently weak in raw text also remains an open question.
Summary updated