Abstract
The Attention Gated Transformation (AGT) network is a novel deep learning architecture designed to explicitly demonstrate hierarchical compositionality in natural language processing. By employing layer-specific attention mechanisms to selectively "pull" information from input text, the model constructs meaning incrementally—from individual words in lower layers to complex phrases and full sentences in higher layers. Evaluated on the Stanford Sentiment Treebank, the AGT network achieves competitive accuracy while providing a transparent, interpretable internal reasoning process.
Introduction
In computer vision, deep networks exhibit clear hierarchical compositionality, where lower layers detect simple features like edges and higher layers recognize complex objects. In contrast, standard natural language processing (NLP) models like LSTMs or CNNs often operate as "black boxes" on abstract embeddings, making their internal reasoning difficult to decipher. This research introduces the Attention Gated Transformation (AGT) network to bridge this gap, creating a model that not only performs well on classification tasks but also visually demonstrates how it builds linguistic meaning step-by-step.
Methods
The AGT network architecture is inspired by Highway Networks, utilizing transform gates to control information flow between layers. The key innovation is that at every layer, the network looks back at the original input text using a layer-specific attention mechanism.
- Attention-Driven Selection: Each layer $l$ calculates attention weights for every word in the input sentence. It then creates a "selection vector" ($s^+_l$), which is a weighted sum of the input word embeddings.
- Gated Update: A transform gate ($T_l$) decides how much of this new "attended" information to incorporate into the current state.
- Recursive Composition: The current layer's representation is a combination of the previous layer's state and the new selection vector.
This recursive structure encourages the model to incrementally retrieve and compose words and phrases as it gets deeper.
Experimental Results
The model was evaluated on the Stanford Sentiment Treebank (SST) for the fine-grained 5-class sentiment category task. This benchmark tests a model's ability to handle complex linguistic structures like negation and sarcasm.
Performance
The AGT network achieved a test-set accuracy of 50.5%. This performance is highly competitive with specialized state-of-the-art models such as the Tree-LSTM (51.0%) and high-order CNNs (51.2%), while offering significantly better interpretability.
Visualizing Compositionality
Analysis of the attention mechanisms revealed several key insights:
- Spiky Attention: Unlike many models that "blur" focus, AGT generates very "spiky" attention weights (often close to 0 or 1), indicating hard choices about which specific words to incorporate.
- Hierarchical Growth: The model follows a "words-to-sentences" path. Lower layers prioritize individual words with strong sentiment or functional importance, such as negations ("not", "never"). Middle layers then expand these into meaningful phrases (e.g., "not so much").
- Re-focusing: In the highest layers, after composing the full sentence, the model often shifts focus back to specific, highly informative phrases that dominate the final classification.
Conclusion
The AGT network provides a transparent alternative to traditional deep NLP models by making its compositional logic visible through layer-specific attention. While the model is sensitive to the initialization of its transform gates, its ability to achieve competitive accuracy while maintaining interpretability is a significant step forward. Future work may involve applying this gated transformation approach to natural language generation tasks.
Summary updated