Abstract
Effective text classification requires capturing both local semantic features and global structural dependencies. Traditional models often prioritize one over the other or rely on computationally expensive external parsers. This paper presents the Sandwich Neural Network (SNN), an architecture that integrates these elements by layering a Convolutional Neural Network (CNN) between Long Short-Term Memory (LSTM) layers. This design allows the model to implicitly learn structural relationships without a parser. Furthermore, we introduce a self-attention mechanism to dynamically fuse semantic and structural representations at the instance level. Our approach achieves state-of-the-art results on several benchmarks, including TREC and SST-5, while remaining more efficient than tree-based alternatives.
Introduction
In Natural Language Processing (NLP), representing a sentence involves a trade-off between capturing local n-gram patterns and understanding the overall syntactic structure. Sequence models like standard LSTMs or CNNs are proficient at identifying local keywords but often fail to grasp long-range dependencies. Conversely, structure-based models like Tree-LSTMs attempt to model grammar directly but typically depend on external parsers that are slow and prone to error propagation.
The proposed Sandwich Neural Network (SNN) aims to bridge this gap. By using a "sandwich" architecture, the model captures local semantic features via a CNN layer and global structural dependencies via a subsequent LSTM layer. This allows the network to adapt to the specific needs of each sentence—whether it is driven by specific keywords or complex grammatical frameworks.
Methodology
The SNN architecture consists of three primary layers:
- Initial LSTM Layer: Processes word embeddings to generate context-aware representations.
- Convolutional Layer (The "Filling"): Extracts local semantic features by analyzing n-grams, identifying high-level phrase meanings.
- Secondary LSTM Layer: Processes the CNN output to capture global structural dependencies and long-range relationships between phrases.
Fusion Strategies
A critical component of the SNN is how it merges the local semantic features ($se$) from the CNN and the global structural features ($st$) from the second LSTM. We explored three strategies:
- Static Weights: Assigns fixed, equal importance to both representations.
- Adaptive Learning: Learns a single set of weights across the entire corpus.
- Self-Attention (SA-SNN): Dynamically calculates weights for each specific sentence, allowing the model to prioritize semantics or structure based on the input's complexity.
Experimental Results
The SA-SNN model was evaluated on four diverse datasets: Movie Review (MR), Stanford Sentiment Treebank (SST-5), TREC question classification, and Subjectivity (SUBJ).
| Dataset | Accuracy |
|---|---|
| TREC | 96.0% |
| MR | 82.1% |
| SST-5 | 50.4% |
| SUBJ | 93.9% |
The model consistently outperformed or matched top-tier models like Tree-LSTMs and DSCNNs. Notably, the SNN achieved these results without the overhead of a parser, making it significantly more efficient for large-scale applications.
Discussion and Conclusion
Analysis of the self-attention weights revealed a strong correlation between syntactic complexity and the importance of the structural representation. As sentences become longer and more complex, the SA-SNN automatically assigns higher weights to the global structure features. This confirms the model's ability to "know" when grammar matters more than individual keywords.
While the SNN is highly effective, future work could involve incorporating external knowledge bases to improve training speed and accuracy. The current model demonstrates that a parser-free, layered architecture can successfully balance the dual requirements of semantics and structure in text classification.
Reference: Zhao, J., et al. (2018). Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification. Proceedings of COLING 2018.
Summary updated