Discriminative Neural Sentence Modeling by Tree-Based Convolution

Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, Zhi Jin

Introduction

In natural language processing, sentence modeling is the fundamental task of converting a sentence into a fixed-length vector representation that captures its semantic meaning. Historically, two primary architectures have dominated this space: Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs).

CNNs are highly effective at extracting local features, such as n-grams, through sliding windows. They are computationally efficient because the "path" from an input word to the output is short, facilitating stable training. However, CNNs are inherently "flat" and ignore the hierarchical, syntactic structure of language. Conversely, RNNs process sentences along their parse trees, composing meanings from the bottom up. While structurally sound, RNNs suffer from "long propagation paths." In deep trees, information from leaf nodes (the actual words) can be diluted by the time it reaches the root, making them difficult to optimize.

Comparison of CNN, RNN, and TBCNN architectures.

To bridge this gap, the authors propose the Tree-Based Convolutional Neural Network (TBCNN). This architecture exploits sentence structure like an RNN while maintaining the efficient, short propagation paths characteristic of a CNN.

Methods: Tree-Based Convolution

The core innovation of TBCNN is the application of convolutional filters to a "subtree" window that slides over a parse tree, rather than a linear window over a sequence. This allows the model to capture structural features regardless of their depth in the tree.

Architectural Variants

The paper explores two primary variants of the TBCNN architecture based on different parsing strategies:

  1. c-TBCNN (Constituency-based): This variant utilizes constituency trees, which group words into nested phrases (e.g., Noun Phrases). Because internal nodes in these trees do not have pre-defined embeddings, the authors employ a pre-training step using a standard recursive neural network to initialize these nodes before applying the convolution.
  2. d-TBCNN (Dependency-based): This variant uses dependency trees, which map grammatical relationships between words (e.g., a subject's relation to a verb). To handle nodes with a varying number of children, d-TBCNN uses a weight-sharing scheme based on the dependency type (e.g., nsubj, dobj) rather than the child's absolute position.
Illustration of feature extraction in constituency-based (c-TBCNN) and dependency-based (d-TBCNN) models.

Pooling Heuristics

After the convolution layer extracts structural features, a Max Pooling layer aggregates them into a fixed-size vector. The authors propose several pooling heuristics to ensure that spatial information within the tree is preserved:

  • Global Pooling: Takes the maximum value across the entire tree.
  • 3-slot Pooling: Divides the tree into "Top," "Lower Left," and "Lower Right" regions to capture coarse-grained positional information.
  • k-slot Pooling: A more flexible approach that segments the tree into $k$ slots based on word order.

Key Insights and Results

TBCNN offers several advantages over traditional sequence-based models. By treating every node in the tree as only one convolutional step away from a feature detector, the model effectively "flattens" the depth of the tree for the optimizer, mitigating the vanishing gradient problems common in deep recursive models.

Performance Benchmarks

The models were evaluated on two classic NLP benchmarks: the Stanford Sentiment Treebank (SST) and the TREC 10 question classification task.

  • Sentiment Analysis: d-TBCNN achieved 51.4% accuracy on the 5-class SST task, outperforming the then-state-of-the-art Tree LSTM (51.0%) and significantly beating standard CNNs and RNNs. Notably, TBCNN maintained high accuracy even as sentence length increased, whereas standard RNNs typically see a performance drop.
  • Question Classification: d-TBCNN reached 96.0% accuracy on TREC 10. This marked the first time a neural network model outperformed traditional methods based on heavy manual feature engineering and hand-coded rules (which peaked at 95.0%).

Conclusion

TBCNN demonstrates that sentence structure is a critical signal for discriminative tasks. By combining the structural awareness of trees with the optimization advantages of convolution, the model achieves superior performance and robustness. Future work may involve moving away from fixed external parsers toward dynamic, learned structures and more sophisticated pooling mechanisms.

Summary updated

Abstract

This paper proposes a tree-based convolutional neural network (TBCNN) for discriminative sentence modeling. Our model leverages either constituency trees or dependency trees of sentences. The tree-based convolution process extracts sentences structural features, which are then aggregated by max pooling. Such architecture allows short propagation paths between the output layer and underlying feature detectors, enabling effective structural feature learning and extraction. We evaluate our models on two tasks: sentiment analysis and question classification. In both experiments, TBCNN outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering. We also make efforts to visualize the tree-based convolution process, shedding light on how our models work.
BibTeX
@inproceedings{mou2015discriminative,
  author = {Lili Mou and Hao Peng and Ge Li and Yan Xu and Lu Zhang and Zhi Jin},
  title = {{Discriminative Neural Sentence Modeling by Tree-Based Convolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1279},
}
Figure