Self-Adaptive Hierarchical Sentence Model

Han Zhao, Zhengdong Lu, Pascal Poupart

Abstract

The ability to accurately model a sentence at varying stages—from words to phrases to the full sentence—is a central challenge in natural language processing. We present the Self-Adaptive Hierarchical Sentence Model (AdaSent), a framework that effectively forms a hierarchy of representations through recursive gated local composition of adjacent segments. By employing a competitive mechanism via gating networks, AdaSent allows representations from multiple levels of the hierarchy to contribute to the final learning task. This architecture effectively mitigates the gradient vanishing problem common in recursive models. Qualitative and quantitative analyses demonstrate that AdaSent automatically selects task-appropriate representations, yielding superior classification performance on five benchmark datasets.

Introduction

In Natural Language Processing (NLP), representing a sentence as a mathematical vector is a fundamental task. Traditional approaches often fall into two extremes: "bag-of-words" models that ignore structural order, or complex recursive trees that rely on rigid, pre-defined grammars. AdaSent proposes a middle ground that is both flexible and powerful, building a multi-scale representation that captures the nuances of language without requiring external parse trees.

Most deep learning models aim to compress a variable-length sentence into a single, fixed-length vector. However, this often leads to information loss, where intermediate phrase-level meanings are washed out, and rigidity, where the model cannot adapt its level of abstraction to the specific task (e.g., sentiment analysis vs. question classification).

The AdaSent Architecture

AdaSent is inspired by Gated Recursive Convolutional Neural Networks but introduces a "self-adaptive" mechanism to manage information flow across a pyramid structure.

1. The Pyramid Structure

For a sentence of length $T$, the model constructs a pyramid of $T$ levels. Level 1 contains individual word vectors. Each subsequent level $t$ is formed by composing adjacent segments from level $t-1$. Consequently, level 2 represents all 2-word phrases, level 3 represents 3-word phrases, and the final level $T$ provides a single vector representing the entire sentence.

The AdaSent architecture: A pyramid of gated compositions pooled into a multi-level representation, weighted by a Gating Network for final classification.

2. Gated Local Composition

The transition between levels utilizes a gating mechanism. To form a representation at level $t$, the model calculates a weighted sum of the left child, the right child, and a non-linear combination of both. This allows the model to either "pass through" important words unchanged or merge them into a new phrase representation, creating a "constant error carousel" similar to LSTMs that helps mitigate gradient vanishing.

3. The Gating Network

After building the pyramid, AdaSent pools information from every level into a global representation $\mathcal{H}$. A separate Gating Network assigns a "belief score" to each level. This allows the model to adaptively weight different scales of the sentence. For short, punchy phrases, the network may favor lower levels; for complex sentences, it may prioritize the top-level global representation.

Experimental Results

AdaSent was evaluated on five benchmark datasets: MR (Movie Reviews), CR (Customer Reviews), SUBJ (Subjectivity), MPQA (Opinion Polarity), and TREC (Question Classification).

  • Superior Accuracy: AdaSent achieved state-of-the-art results on 4 out of 5 datasets.
  • MPQA Breakthrough: On the MPQA dataset, which involves very short snippets, AdaSent reached 93.3% accuracy, significantly outperforming standard CNNs (89.6%).
  • Parameter Efficiency: The model achieves these results with significantly fewer parameters (10K–100K) compared to large-scale CNNs that often exceed 400K parameters.

Qualitative Analysis

Visualizing the learned representations reveals how AdaSent effectively clusters different categories. The gating mechanism ensures that the most discriminative features, whether at the word or phrase level, are preserved for the classifier.

PCA/t-SNE visualizations of sentence representations across different datasets, showing the model's ability to form distinct clusters for classification tasks.

Conclusion

AdaSent demonstrates that a self-adaptive, hierarchical approach to sentence modeling can overcome the limitations of fixed-length representations. By allowing the task to dictate the relevant level of abstraction, the model provides a robust and efficient framework for text classification. Future work may explore extending this multi-scale representation to more complex tasks such as machine translation and document summarization.

Summary updated

Abstract

The ability to accurately model a sentence at varying stages (e.g., word-phrase-sentence) plays a central role in natural language processing. As an effort towards this goal we propose a self-adaptive hierarchical sentence model (AdaSent). AdaSent effectively forms a hierarchy of representations from words to phrases and then to sentences through recursive gated local composition of adjacent segments. We design a competitive mechanism (through gating networks) to allow the representations of the same sentence to be engaged in a particular learning task (e.g., classification), therefore effectively mitigating the gradient vanishing problem persistent in other recursive models. Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.
BibTeX
@inproceedings{zhao2015self,
  author = {Han Zhao and Zhengdong Lu and Pascal Poupart},
  title = {{Self-Adaptive Hierarchical Sentence Model}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
  url = {https://openreview.net/forum?id=SkZnPNzubH},
}
Figure