When Are Tree Structures Necessary for Deep Learning of Representations?

Jiwei Li, Minh-Thang Luong, Dan Jurafsky, Eudard Hovy

TL;DR

This paper presents a rigorous, apples-to-apples benchmark comparing tree-structured recursive neural networks against sequential recurrent models (like LSTMs) across four distinct NLP tasks. The authors find that explicit tree structures are only strictly necessary for tasks requiring long-distance semantic dependencies between headwords; for most other tasks, sequential models perform just as well, especially when enhanced by a simple punctuation-based clausal segmentation strategy.

What Problem Does This Paper Address?

Deep learning in NLP relies heavily on compositional models to aggregate individual word embeddings into fixed-dimensional vectors representing phrases, sentences, or documents. Historically, these architectures have fallen into two main camps:

  1. Recurrent models (Sequence models): Process text sequentially from left to right (or bidirectionally), considering word order but ignoring hierarchical linguistic structure.
  2. Recursive models (Tree models): Combine neighboring words bottom-up based on the hierarchical structure of a syntactic parse tree.

While recursive models are intuitively appealing because they mirror human grammar, they come with a heavy tax: they rely on syntactic parsers that are computationally slow, domain-dependent, and prone to propagating errors. The authors address a fundamental question: Is the added complexity of syntactic parse trees actually necessary, or can sequential models achieve comparable performance when evaluated fairly?

Methods

The authors benchmark standard and LSTM-based variants of both recursive and recurrent architectures. To ensure a fair comparison, they enforce identical training frameworks (AdaGrad optimization, initialization strategies, and regularization).

Beyond standard sequential and tree models, they introduce a novel, simple approximation to bridge the gap:

  • Deep Bi-LSTM Hierarchical Sequence Model: Instead of parsing a sentence syntactically, this model uses punctuation (commas, periods, question marks) to slice long sentences into clause-like units. Each sub-sentence is processed separately by a sequence-level LSTM, and a higher-level LSTM then combines these clausal representations.
Illustration of standard vs. hierarchical sequence models

Datasets

The models are evaluated across five datasets spanning four representative NLP tasks:

  1. Stanford Sentiment Treebank (SST): Sentiment classification containing gold-standard labels for every single parse tree constituent (phrases and full sentences).
  2. Pang et al. (2002) Sentiment Dataset: Binary sentence-level sentiment classification where supervision is only provided globally at the end of long sequences.
  3. UMD-QA Dataset: A factoid question-answering dataset framed as matching question sentences to target answer phrases.
  4. SemEval-2010 Task 8: Semantic relation classification between pairs of nominals (e.g., identifying a component-whole relationship).
  5. RST Discourse Treebank (RST-DT): Discourse parsing to identify rhetorical relations between elementary discourse units (EDUs).

Key Insights and Ideas

  • Syntax Shines at a Distance: Tree structures provide a significant advantage primarily when a task requires connecting words that are structurally close but textually far apart (e.g., a verb and its distant direct object). In sequential models, the network must "remember" the first word through many time steps, leading to information loss.
  • Punctuation is a Cheap Proxy for Syntax: For sentiment analysis on long sentences, the main benefit of a parse tree isn't its intricate internal grammar rules, but rather its ability to break the sentence into manageable chunks. The authors' hierarchical sequence model achieves parity with Tree-LSTMs simply by splitting text at punctuation marks.
  • Sequence Fragments Carry Implicit Structure: In question-answering matching, intermediate hidden states of sequential models embed informative evidence just as effectively as linguistically meaningful parse tree nodes. This indicates that LSTMs can implicitly capture compositionality without explicit trees.

Key Results

Sentiment Analysis (SST)

On fine-grained sentence-level classification, the Tree LSTM achieved 50.4% accuracy, slightly outperforming a standard Bi-Sequence LSTM (49.8%). However, the Hierarchical Sequence model successfully bridged this gap, achieving 50.7% accuracy by simply leveraging punctuation boundaries.

Figure 1: Transforming Stanford Sentiment Treebank to Sequences for Sequence Models.

Semantic Relation Classification

Tree models decisively outperformed sequential models here. The standard tree model achieved 74.8% accuracy compared to the standard sequence model's 71.2% ($p = 0.004$), confirming that explicit trees are highly valuable when target words are separated by long textual distances.

Figure 4: Illustration of Models for Semantic Re- lationship Classification.

Discourse Parsing & QA

No large or statistically significant differences were found between equivalent recursive and recurrent models on the RST dataset or the UMD-QA dataset, suggesting that sequence models are entirely sufficient for these tasks.

Limitations and Future Work

The authors note a few important caveats:

  • Model Complexity: The study focuses primarily on basic, single-layer neural compositions to ensure a clean comparison. It remains unclear if these insights hold for highly complex, multi-layer state-of-the-art architectures.
  • Training Uniformity: Forcing identical training configurations (e.g., optimization algorithms) might inadvertently disadvantage certain models that thrive under specific hyperparameter tuning.
  • Future Directions: A promising direction is exploring whether sequential models can discover implicit recursive structures in tasks with massive structural reordering, such as Chinese-to-English machine translation.

Summary updated

Abstract

Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. But there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper we benchmark {\bf recursive} neural models against sequential {\bf recurrent} neural models (simple recurrent and LSTM models), enforcing apples-to-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment classification at the sentence level and phrase level; (2) matching questions to answer-phrases; (3) discourse parsing; (4) semantic relation extraction (e.g., {\em component-whole} between nouns). Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like semantic relation extraction) that require associating headwords across a long distance, particularly on very long sequences. We then introduce a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models.
BibTeX
@inproceedings{li2015when,
  author = {Jiwei Li and Minh-Thang Luong and Dan Jurafsky and Eudard Hovy},
  title = {{When Are Tree Structures Necessary for Deep Learning of Representations?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1278},
}
Figure