Abstract
Standard sequential models often overlook the hierarchical nature of language. While Recursive Neural Networks (RvNNs) incorporate syntax, they typically rely on expensive annotated data. This paper presents an unsupervised Tree-LSTM augmented with a structural attention mechanism. By weighting intermediate nodes based on task relevance, the model discovers parse trees that are both more linguistically accurate and more explainable, achieving state-of-the-art performance on key NLP benchmarks.
Introduction
The inherent hierarchical structure of language is a fundamental aspect of linguistics that sequential models like LSTMs often fail to capture explicitly. Recursive Neural Networks (RvNNs) were designed to process text according to these structures, but their reliance on "gold standard" parse trees—which require expert annotation—limits their scalability. Unsupervised models, such as the Gumbel Tree-LSTM, attempt to learn these structures end-to-end. However, these models frequently produce "junk" trees—highly skewed or nonsensical structures—that minimize task loss without reflecting true grammatical relationships. This work introduces a structural attention mechanism to align task-driven learning with linguistic explainability.
Methodology
The core innovation is the Structural Attentive Layer integrated into the Gumbel Tree-LSTM framework.
The model utilizes the Gumbel-softmax trick to maintain differentiability while making discrete tree-building choices. Unlike traditional Tree-LSTMs that treat all nodes equally in the final representation, the proposed attention mechanism maps all nodes into a shared embedding space to calculate "validity scores." The final sentence representation is a weighted sum of these nodes, encouraging the model to focus on the most salient hierarchical components.
Key Insights and Discussion
A significant finding is that attention serves as a powerful regularizer for tree induction. By identifying which nodes contribute most to the final classification, the model naturally avoids the "left-branching" bias common in previous unsupervised methods, instead producing more balanced and linguistically sensible trees.
Qualitative analysis indicates that the learned embeddings are more discriminative than those from standard word2vec or basic Gumbel Tree-LSTMs. Furthermore, the model's induced trees align more closely with human grammar, as evidenced by improved F1 scores against gold-standard parsers.
Evaluation and Results
The model was benchmarked on SNLI, SST, and QQP datasets. Key results include:
- Task Performance: Achieved 85.9% accuracy on SNLI and an 85.4 Macro-F1 on QQP, outperforming standard Tree-LSTM baselines.
- Linguistic Fidelity: Attained a 31.3 F1 score against the Stanford Parser, a significant jump from the 25.2 achieved by the base Gumbel Tree-LSTM.
- Structural Complexity: The average tree depth of 4.7 is much closer to human-annotated trees (5.7) than prior unsupervised models.
Conclusion
The integration of structural attention into unsupervised Tree-LSTMs significantly enhances the explainability and grammatical accuracy of induced parse trees. While raw accuracy gains are modest, the improvement in model transparency and linguistic alignment offers a promising path for more robust and interpretable NLP systems.
Summary updated