On Learning Interpreted Languages with Recurrent Models

Denis PapernoComput. Linguistics 2022 · OpenReview

TL;DR

This paper investigates whether recurrent neural networks (RNNs) can learn to understand compositional language, a key aspect of human language. They found that while RNNs can generalize to compositional interpretation under specific, favorable conditions (like a curriculum and left-branching structures), they struggle with right-branching structures, suggesting limitations in their ability to truly grasp recursive syntax and semantics.

What Problem Does This Paper Address?

A major challenge in Natural Language Processing (NLP) is achieving "compositional generalization." This means that models should not just memorize specific phrases but understand how to combine the meanings of words and phrases to interpret novel, complex expressions. Current NLP models often rely on shallow cues and struggle to generalize beyond their training data, failing to exhibit the deeper understanding seen in humans. This paper aims to bridge this gap by testing if RNNs, inspired by human sequential processing, can learn compositional interpretation from simplified, artificial languages that mimic core properties of natural language, specifically recursion and compositionality.

Methods

The authors employ standard recurrent neural network architectures: Simple Recurrent Networks (SRNs), Long Short-Term Memory networks (LSTMs), and Gated Recurrent Units (GRUs). These models are trained on artificially constructed "toy interpreted languages" designed to isolate specific linguistic properties. The core task is to identify the individual referent of a given linguistic expression.

A key methodological innovation is the use of a curriculum learning approach. Instead of presenting all training examples at once, the models are gradually exposed to more complex examples, starting with simpler ones. This mimics how humans might learn language, building from basic elements to more intricate structures. The paper also contrasts left-branching structures (e.g., "Ann's child") with right-branching structures (e.g., "the child of Ann") to test how syntactic order affects learning.

Datasets

The paper introduces novel, simplified datasets based on fragments of English. These datasets are generated from grammars that create either left-branching or right-branching structures. The "toy universe" for interpretation consists of four individuals and four relation nouns (e.g., "child," "parent," "friend," "enemy"). The interpretation of these relations is randomized for each experiment to prevent models from exploiting consistent biases. Datasets are generated for varying "complexity" levels, where complexity is defined by the number of content words in an expression.

Key Insights and Ideas

  • Compositionality is not automatic: Even with powerful RNNs, compositional generalization doesn't emerge effortlessly. It requires specific learning conditions.
  • Curriculum learning is crucial: Gradually introducing complexity significantly improves RNNs' ability to learn recursive structures, suggesting a need to learn basic interpretations before tackling complex composition.
  • Left-branching bias: RNNs show a strong preference and much higher accuracy for left-branching structures. This suggests they learn to process information linearly and sequentially, mirroring the input order, rather than truly understanding nested recursive structures.
  • Memorization vs. Generalization: While models must memorize basic interpretations (complexity 1 and 2 expressions), true generalization to unseen complex expressions is the ultimate goal, and this is where RNNs show limitations.
  • Data hungriness: Achieving good generalization, especially for recursive structures, requires a substantial amount of training data, particularly for the more complex examples.

Key Results

  • High accuracy on left-branching: GRU and LSTM models achieved near-perfect accuracy (around 0.92-0.99) on left-branching structures when trained with a curriculum.
  • Low accuracy on right-branching: In contrast, accuracy on right-branching structures was significantly lower, often below 0.3, even with a curriculum.
  • Curriculum impact: Models trained without a curriculum performed poorly, highlighting its importance. For example, LSTM accuracy on right-branching structures without a curriculum was around 0.23-0.34, compared to 0.88-0.92 with a curriculum.
  • Generalization to unseen lengths: Models trained on complexities 1-3 could generalize to higher complexities (e.g., complexity 7), achieving accuracies up to 0.98 for left-branching structures, but struggled with right-branching ones.

Limitations and Future Work

The paper acknowledges that the artificial languages are simplified and do not capture the full complexity of natural language. A significant limitation is the observed asymmetry: RNNs generalize well to left-branching structures but poorly to right-branching ones. This suggests that current recurrent models, while capable of sequential processing, may not fully grasp the recursive nature of human language.

Future work could explore:

  • Bidirectional RNNs: Investigating whether bidirectional models can overcome the left-branching asymmetry.
  • Alternative architectures: Examining models with explicit stack memory or chart parsing components, which are better suited for hierarchical structures.
  • Attention mechanisms: Exploring attention-based models to see if they offer better generalization.
  • Human language learning comparison: Investigating whether human learners also exhibit a bias towards left-branching structures and require substantial data to learn recursion.

Summary updated

BibTeX
@inproceedings{paperno2022learning,
  author = {Denis Paperno},
  title = {{On Learning Interpreted Languages with Recurrent Models}},
  booktitle = {Comput. Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=0QhB7K89EQ},
}
Figure