Self-Adaptive Hierarchical Sentence Model

Han Zhao, Zhengdong Lu, Pascal Poupart

Abstract

The ability to accurately model a sentence at varying stages (e.g., word-phrase-sentence) plays a central role in natural language processing. As an effort towards this goal we propose a self-adaptive hierarchical sentence model (AdaSent). AdaSent effectively forms a hierarchy of representations from words to phrases and then to sentences through recursive gated local composition of adjacent segments. We design a competitive mechanism (through gating networks) to allow the representations of the same sentence to be engaged in a particular learning task (e.g., classification), therefore effectively mitigating the gradient vanishing problem persistent in other recursive models. Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.

Introduction

Representing a sentence as a continuous vector is a foundational step for modern natural language processing (NLP) tasks like classification, sentiment analysis, and machine translation. However, traditional architectures face significant hurdles. Bag-of-Words (cBoW) models ignore structural nuances, while Recurrent Neural Networks (RNNs) often suffer from vanishing gradients and a bias toward the end of sequences. Recursive Neural Networks attempt to capture hierarchy but typically rely on rigid, pre-defined parse trees that are prone to external errors.

Furthermore, most existing models compress all compositional information into a single, fixed-length vector at the final layer. This "bottleneck" often loses valuable intermediate details—such as the specific meaning of short phrases—that might be critical for certain tasks. AdaSent (Self-Adaptive Hierarchical Sentence Model) addresses these limitations by building a multi-scale pyramid of representations and dynamically selecting the most relevant level of abstraction.

The AdaSent Architecture

AdaSent constructs a directed acyclic graph (DAG) pyramid of representations, moving from individual words to the entire sentence through an ensemble-like approach.

The AdaSent architecture, illustrating the pyramid of representations, level pooling, and the global gating mechanism that weights predictions from different levels of abstraction.

Recursive Gated Local Composition

The model builds the pyramid level by level. The base level consists of word vectors. To form the next level, AdaSent uses a local gating mechanism (a softmax function) to decide if a phrase's meaning should be derived from its left child, its right child, or a non-linear combination of both. This gated approach acts similarly to the "forget gate" in an LSTM, creating linear paths for gradients to flow backward and effectively mitigating the vanishing gradient problem.

Level Pooling and Global Gating

At each level $t$ of the pyramid, the model pools representations to create a summary vector for that specific level of abstraction. Instead of relying solely on the final top-level vector, AdaSent feeds every level's summary vector into a classifier. A global gating network assigns a "belief score" to each level, and the final prediction is a weighted mixture of all levels. This implicit ensembling makes the model highly robust to local noise compared to standard sequential models.

Experimental Evaluation

The authors evaluated AdaSent on five standard benchmark datasets: Movie Reviews (MR), Customer Reviews (CR), Subjectivity (SUBJ), Phrase-level opinion polarity (MPQA), and Question classification (TREC).

Key Results

  • Superior Accuracy: AdaSent consistently outperformed competitors such as cBoW, standard RNNs, and Paragraph Vectors. It achieved notable scores of 83.1% on MR and 95.5% on SUBJ.
  • Short-Phrase Dominance: On the MPQA dataset, which focuses on short phrases, AdaSent achieved 93.3% accuracy, significantly outperforming the next best model (BRNN at 90.3%). This success is attributed to its explicit multi-scale pooling.
  • Parameter Efficiency: Remarkably, AdaSent achieved these results using only 10K to 100K parameters, whereas contemporary Convolutional Neural Networks (CNNs) often required upwards of 400K parameters.

Adaptive Representation Selection

A core strength of AdaSent is its ability to learn which level of abstraction is most useful for a given task. Qualitative analysis shows that for tasks involving short phrases (like MPQA), the model relies more heavily on lower-level representations. For complex sentence-level tasks, it shifts its focus to higher levels of the pyramid.

Visualization of the global gating network's belief scores across different levels, demonstrating how the model dynamically selects task-relevant abstractions during inference.

Conclusion and Future Work

AdaSent introduces a flexible, hierarchical approach to sentence modeling that avoids the pitfalls of fixed-length bottlenecks and rigid parse trees. While the model successfully addresses vanishing gradients, the authors noted that exploding gradients remain a challenge, requiring techniques like gradient norm-clipping and Frobenius norm regularization. Future research may extend this multi-scale hierarchical representation to more complex sequence tasks, such as machine translation or semantic matching, where intermediate phrase representations are particularly valuable.

Summary updated

Abstract

The ability to accurately model a sentence at varying stages (e.g., word-phrase-sentence) plays a central role in natural language processing. As an effort towards this goal we propose a self-adaptive hierarchical sentence model (AdaSent). AdaSent effectively forms a hierarchy of representations from words to phrases and then to sentences through recursive gated local composition of adjacent segments. We design a competitive mechanism (through gating networks) to allow the representations of the same sentence to be engaged in a particular learning task (e.g., classification), therefore effectively mitigating the gradient vanishing problem persistent in other recursive models. Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.
BibTeX
@inproceedings{zhao2015self,
  author = {Han Zhao and Zhengdong Lu and Pascal Poupart},
  title = {{Self-Adaptive Hierarchical Sentence Model}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
  url = {https://openreview.net/forum?id=SkZnPNzubH},
}
Figure