Abstract
The ability to accurately model a sentence at varying stages—from words to phrases to the full sentence—is a central challenge in natural language processing. We present the Self-Adaptive Hierarchical Sentence Model (AdaSent), a framework that effectively forms a hierarchy of representations through recursive gated local composition of adjacent segments. By employing a competitive mechanism via gating networks, AdaSent allows representations from multiple levels of the hierarchy to contribute to the final learning task. This architecture effectively mitigates the gradient vanishing problem common in recursive models. Qualitative and quantitative analyses demonstrate that AdaSent automatically selects task-appropriate representations, yielding superior classification performance on five benchmark datasets.
Introduction
In Natural Language Processing (NLP), representing a sentence as a mathematical vector is a fundamental task. Traditional approaches often fall into two extremes: "bag-of-words" models that ignore structural order, or complex recursive trees that rely on rigid, pre-defined grammars. AdaSent proposes a middle ground that is both flexible and powerful, building a multi-scale representation that captures the nuances of language without requiring external parse trees.
Most deep learning models aim to compress a variable-length sentence into a single, fixed-length vector. However, this often leads to information loss, where intermediate phrase-level meanings are washed out, and rigidity, where the model cannot adapt its level of abstraction to the specific task (e.g., sentiment analysis vs. question classification).
The AdaSent Architecture
AdaSent is inspired by Gated Recursive Convolutional Neural Networks but introduces a "self-adaptive" mechanism to manage information flow across a pyramid structure.
1. The Pyramid Structure
For a sentence of length $T$, the model constructs a pyramid of $T$ levels. Level 1 contains individual word vectors. Each subsequent level $t$ is formed by composing adjacent segments from level $t-1$. Consequently, level 2 represents all 2-word phrases, level 3 represents 3-word phrases, and the final level $T$ provides a single vector representing the entire sentence.
2. Gated Local Composition
The transition between levels utilizes a gating mechanism. To form a representation at level $t$, the model calculates a weighted sum of the left child, the right child, and a non-linear combination of both. This allows the model to either "pass through" important words unchanged or merge them into a new phrase representation, creating a "constant error carousel" similar to LSTMs that helps mitigate gradient vanishing.
3. The Gating Network
After building the pyramid, AdaSent pools information from every level into a global representation $\mathcal{H}$. A separate Gating Network assigns a "belief score" to each level. This allows the model to adaptively weight different scales of the sentence. For short, punchy phrases, the network may favor lower levels; for complex sentences, it may prioritize the top-level global representation.
Experimental Results
AdaSent was evaluated on five benchmark datasets: MR (Movie Reviews), CR (Customer Reviews), SUBJ (Subjectivity), MPQA (Opinion Polarity), and TREC (Question Classification).
- Superior Accuracy: AdaSent achieved state-of-the-art results on 4 out of 5 datasets.
- MPQA Breakthrough: On the MPQA dataset, which involves very short snippets, AdaSent reached 93.3% accuracy, significantly outperforming standard CNNs (89.6%).
- Parameter Efficiency: The model achieves these results with significantly fewer parameters (10K–100K) compared to large-scale CNNs that often exceed 400K parameters.
Qualitative Analysis
Visualizing the learned representations reveals how AdaSent effectively clusters different categories. The gating mechanism ensures that the most discriminative features, whether at the word or phrase level, are preserved for the classifier.
Conclusion
AdaSent demonstrates that a self-adaptive, hierarchical approach to sentence modeling can overcome the limitations of fixed-length representations. By allowing the task to dictate the relevant level of abstraction, the model provides a robust and efficient framework for text classification. Future work may explore extending this multi-scale representation to more complex tasks such as machine translation and document summarization.
Summary updated