Deep Recursive Neural Networks for Compositionality in Language

Ozan Irsoy, Claire CardieNeurIPS 2014 · OpenReview

In the world of Natural Language Processing (NLP), we often talk about "depth." Usually, this refers to stacking layers in a feed-forward network or the unrolling of a recurrent network over time. However, language has a third kind of depth: the structural depth of a parse tree.

In this paper, Ozan İrsoy and Claire Cardie from Cornell University introduce Deep Recursive Neural Networks (DRNNs), a model that combines structural recursion with stacked hierarchical layers to better capture how meaning is composed in language.

TL;DR

The authors propose stacking multiple recursive neural network layers to create a "Deep Recursive Neural Network." By combining structural depth (following a parse tree) with representational depth (stacking layers), the model achieves state-of-the-art results on fine-grained sentiment analysis, outperforming previous recursive and convolutional baselines.

What Problem Does This Paper Address?

Language is inherently compositional—we build meaning by combining words into phrases, and phrases into sentences. Recursive Neural Networks (RecNNs) were designed to model this by applying the same neural transformation at every branch of a sentence's parse tree.

However, traditional RecNNs have a limitation: they are "shallow" in terms of abstraction. While they are deep in structure (the tree might be many levels high), they use the same set of weights at every node. This is unlike deep feed-forward or recurrent networks, where stacking layers allows the model to learn a hierarchy of features—from low-level edges to high-level concepts. The authors set out to bridge this gap: can we add "depth in space" (stacked layers) to the "depth in structure" (the tree) of a recursive network?

Methods

The core innovation is the Deep Recursive Neural Network (DRNN). Instead of a node in a parse tree having a single hidden state, it has multiple hidden states stacked vertically.

Illustration of a Deep Recursive Neural Network
  1. Stacked Layers: For any node $\eta$ in the tree, the hidden state at layer $i$ is calculated using the hidden states of its children at the same layer $i$, plus its own hidden state from the previous layer ($i-1$). This allows higher layers to represent more abstract semantic information.
  2. Untying Leaves and Internals: Standard RecNNs often use the same weights for word-level inputs and phrase-level compositions. The authors "untie" these, using a specific transformation for leaf nodes (words) and another for internal nodes (phrases). This allows the model to handle high-dimensional word embeddings without a quadratic explosion in parameters.
  3. Modern Architecture Choices: The model uses Rectified Linear Units (ReLUs) instead of the traditional $tanh$ activation, which helps mitigate vanishing gradients in deep structures. They also utilize Dropout across the layers for regularization.

Datasets

The models were evaluated on the Stanford Sentiment Treebank (SST).

  • Scale: 11,855 sentences, broken down into 215,154 unique phrases.
  • Granularity: Unlike many datasets that only label the full sentence, SST provides labels for every single node in the parse tree.
  • Task: 5-class fine-grained sentiment (Very Negative to Very Positive) and binary classification.
  • Embeddings: They used publicly available 300-dimensional Word2Vec vectors trained on Google News.

Key Insights and Ideas

  • Depth vs. Width: The authors found that increasing depth is more effective than increasing width. A 4-layer DRNN with fewer hidden units per layer outperformed a 1-layer RecNN with many more units, even when the total number of parameters was kept constant.
  • The "Aha" Moment in Abstraction: Qualitative analysis showed that different layers "see" the sentence differently.
  • Layer 1 acts like a syntactic feature detector; it might see "good" and "bad" as similar because they are both adjectives.
  • Layer 3 (the top layer) is where the sentiment "clicks." It clearly separates "good" and "bad" into different semantic spaces, showing that the hierarchy is successfully distilling meaning.
  • Handling Negation: The multi-layer approach proved better at handling complex linguistic phenomena like negation (e.g., "not great"), which often trips up shallower models.

Key Results

The DRNN set a new state-of-the-art for sentiment analysis at the time of publication:

Test accuracies on the Stanford Sentiment Treebank
  • Fine-grained Accuracy: The 4-layer DRNN achieved 49.8%, significantly beating the Recursive Neural Tensor Network (RNTN) at 45.7% and Paragraph Vectors at 48.7%.
  • Binary Accuracy: It achieved 86.6%, performing competitively with the best convolutional models of the era.
  • Efficiency: The "untied" weight scheme allowed the model to use large 300D word vectors while keeping the internal hidden layers relatively small (e.g., 174 units), making the model more parameter-efficient.

Limitations and Future Work

  • Fixed Word Vectors: The authors kept word embeddings fixed to simplify the experiment. Fine-tuning these embeddings during training would likely yield even higher performance.
  • Dependency on Parse Trees: The model requires a high-quality parse tree as input. In languages or domains where parsers are inaccurate, the model's performance would likely suffer.
  • Supervision: The model relies on the rich, node-level supervision of the SST. The authors suggest that exploring unsupervised or semi-supervised versions of DRNNs (similar to autoencoders) would be a valuable next step for broader NLP applications.

Summary updated

Abstract

Recursive neural networks comprise a class of architecture that can operate on structured input. They have been previously successfully applied to model compositionality in natural language using parse-tree-based structural representations. Even though these architectures are deep in structure, they lack the capacity for hierarchical representation that exists in conventional deep feed-forward networks as well as in recently investigated deep recurrent neural networks. In this work we introduce a new architecture --- a deep recursive neural network (deep RNN) --- constructed by stacking multiple recursive layers. We evaluate the proposed model on the task of fine-grained sentiment classification. Our results show that deep RNNs outperform associated shallow counterparts that employ the same number of parameters. Furthermore, our approach outperforms previous baselines on the sentiment analysis task, including a multiplicative RNN variant as well as the recently introduced paragraph vectors, achieving new state-of-the-art results. We provide exploratory analyses of the effect of multiple layers and show that they capture different aspects of compositionality in language.
BibTeX
@inproceedings{irsoy2014deep,
  author = {Ozan Irsoy and Claire Cardie},
  title = {{Deep Recursive Neural Networks for Compositionality in Language}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=HyWO6L-_bH},
}
Figure