Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification

Jianyu Zhao, Zhiqiang Zhan, Qichuan Yang, Yang Zhang, Changjian Hu, Zhensheng Li, Liuxin Zhang, Zhiqiang HeCOLING 2018 · OpenReview

Abstract

Effective text classification requires capturing both local semantic features and global structural dependencies. Traditional models often prioritize one over the other or rely on computationally expensive external parsers. This paper presents the Sandwich Neural Network (SNN), an architecture that integrates these elements by layering a Convolutional Neural Network (CNN) between Long Short-Term Memory (LSTM) layers. This design allows the model to implicitly learn structural relationships without a parser. Furthermore, we introduce a self-attention mechanism to dynamically fuse semantic and structural representations at the instance level. Our approach achieves state-of-the-art results on several benchmarks, including TREC and SST-5, while remaining more efficient than tree-based alternatives.

Introduction

In Natural Language Processing (NLP), representing a sentence involves a trade-off between capturing local n-gram patterns and understanding the overall syntactic structure. Sequence models like standard LSTMs or CNNs are proficient at identifying local keywords but often fail to grasp long-range dependencies. Conversely, structure-based models like Tree-LSTMs attempt to model grammar directly but typically depend on external parsers that are slow and prone to error propagation.

The proposed Sandwich Neural Network (SNN) aims to bridge this gap. By using a "sandwich" architecture, the model captures local semantic features via a CNN layer and global structural dependencies via a subsequent LSTM layer. This allows the network to adapt to the specific needs of each sentence—whether it is driven by specific keywords or complex grammatical frameworks.

Methodology

The SNN architecture consists of three primary layers:

  1. Initial LSTM Layer: Processes word embeddings to generate context-aware representations.
  2. Convolutional Layer (The "Filling"): Extracts local semantic features by analyzing n-grams, identifying high-level phrase meanings.
  3. Secondary LSTM Layer: Processes the CNN output to capture global structural dependencies and long-range relationships between phrases.
Figure 1: Overview of the SNN architecture and the various fusion methods employed to combine semantic and structural features.

Fusion Strategies

A critical component of the SNN is how it merges the local semantic features ($se$) from the CNN and the global structural features ($st$) from the second LSTM. We explored three strategies:

  • Static Weights: Assigns fixed, equal importance to both representations.
  • Adaptive Learning: Learns a single set of weights across the entire corpus.
  • Self-Attention (SA-SNN): Dynamically calculates weights for each specific sentence, allowing the model to prioritize semantics or structure based on the input's complexity.
Figure 3: Detailed view of the Self-Attention fusion mechanism, which utilizes word embeddings and MLPs to determine instance-level weights.

Experimental Results

The SA-SNN model was evaluated on four diverse datasets: Movie Review (MR), Stanford Sentiment Treebank (SST-5), TREC question classification, and Subjectivity (SUBJ).

Dataset Accuracy
TREC 96.0%
MR 82.1%
SST-5 50.4%
SUBJ 93.9%

The model consistently outperformed or matched top-tier models like Tree-LSTMs and DSCNNs. Notably, the SNN achieved these results without the overhead of a parser, making it significantly more efficient for large-scale applications.

Discussion and Conclusion

Analysis of the self-attention weights revealed a strong correlation between syntactic complexity and the importance of the structural representation. As sentences become longer and more complex, the SA-SNN automatically assigns higher weights to the global structure features. This confirms the model's ability to "know" when grammar matters more than individual keywords.

While the SNN is highly effective, future work could involve incorporating external knowledge bases to improve training speed and accuracy. The current model demonstrates that a parser-free, layered architecture can successfully balance the dual requirements of semantics and structure in text classification.

Reference: Zhao, J., et al. (2018). Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification. Proceedings of COLING 2018.

Summary updated

BibTeX
@inproceedings{zhao2018adaptive,
  author = {Jianyu Zhao and Zhiqiang Zhan and Qichuan Yang and Yang Zhang and Changjian Hu and Zhensheng Li and Liuxin Zhang and Zhiqiang He},
  title = {{Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=43QxefvB420},
}
Figure