Unsupervised Learning of Explainable Parse Trees for Improved Generalisation

Atul Sahay, Ayush Maheshwari, Ritesh Kumar, Ganesh Ramakrishnan, Manjesh Kumar Hanawal, Kavi Arya

Abstract

Standard sequential models often overlook the hierarchical nature of language. While Recursive Neural Networks (RvNNs) incorporate syntax, they typically rely on expensive annotated data. This paper presents an unsupervised Tree-LSTM augmented with a structural attention mechanism. By weighting intermediate nodes based on task relevance, the model discovers parse trees that are both more linguistically accurate and more explainable, achieving state-of-the-art performance on key NLP benchmarks.

Introduction

The inherent hierarchical structure of language is a fundamental aspect of linguistics that sequential models like LSTMs often fail to capture explicitly. Recursive Neural Networks (RvNNs) were designed to process text according to these structures, but their reliance on "gold standard" parse trees—which require expert annotation—limits their scalability. Unsupervised models, such as the Gumbel Tree-LSTM, attempt to learn these structures end-to-end. However, these models frequently produce "junk" trees—highly skewed or nonsensical structures—that minimize task loss without reflecting true grammatical relationships. This work introduces a structural attention mechanism to align task-driven learning with linguistic explainability.

Methodology

The core innovation is the Structural Attentive Layer integrated into the Gumbel Tree-LSTM framework.

Model architecture overview.

The model utilizes the Gumbel-softmax trick to maintain differentiability while making discrete tree-building choices. Unlike traditional Tree-LSTMs that treat all nodes equally in the final representation, the proposed attention mechanism maps all nodes into a shared embedding space to calculate "validity scores." The final sentence representation is a weighted sum of these nodes, encouraging the model to focus on the most salient hierarchical components.

Key Insights and Discussion

A significant finding is that attention serves as a powerful regularizer for tree induction. By identifying which nodes contribute most to the final classification, the model naturally avoids the "left-branching" bias common in previous unsupervised methods, instead producing more balanced and linguistically sensible trees.

Attention weights on induced parse trees.

Qualitative analysis indicates that the learned embeddings are more discriminative than those from standard word2vec or basic Gumbel Tree-LSTMs. Furthermore, the model's induced trees align more closely with human grammar, as evidenced by improved F1 scores against gold-standard parsers.

Evaluation and Results

The model was benchmarked on SNLI, SST, and QQP datasets. Key results include:

  • Task Performance: Achieved 85.9% accuracy on SNLI and an 85.4 Macro-F1 on QQP, outperforming standard Tree-LSTM baselines.
  • Linguistic Fidelity: Attained a 31.3 F1 score against the Stanford Parser, a significant jump from the 25.2 achieved by the base Gumbel Tree-LSTM.
  • Structural Complexity: The average tree depth of 4.7 is much closer to human-annotated trees (5.7) than prior unsupervised models.

Conclusion

The integration of structural attention into unsupervised Tree-LSTMs significantly enhances the explainability and grammatical accuracy of induced parse trees. While raw accuracy gains are modest, the improvement in model transparency and linguistic alignment offers a promising path for more robust and interpretable NLP systems.

Summary updated

Abstract

Recursive neural networks (RvNN) have been shown useful for learning sentence representations and helped achieve competitive performance on several natural language inference tasks. However, recent RvNN-based models fail to learn simple grammar and meaningful semantics in their intermediate tree representation. In this work, we propose an attention mechanism over Tree-LSTMs to learn more meaningful and explainable parse tree structures. We also demonstrate the superior performance of our proposed model on natural language inference, semantic relatedness, and sentiment analysis tasks and compare them with other state-of-the-art RvNN based methods. Further, we present a detailed qualitative and quantitative analysis of the learned parse trees and show that the discovered linguistic structures are more explainable, semantically meaningful, and grammatically correct than recent approaches. The source code of the paper is available at https://github.com/atul04/Explainable-Latent-Structures-Using-Attention.
BibTeX
@inproceedings{sahay2021unsupervised,
  author = {Atul Sahay and Ayush Maheshwari and Ritesh Kumar and Ganesh Ramakrishnan and Manjesh Kumar Hanawal and Kavi Arya},
  title = {{Unsupervised Learning of Explainable Parse Trees for Improved Generalisation}},
  booktitle = {IJCNN},
  year = {2021},
  doi = {10.1109/ijcnn52387.2021.9533448},
}
Figure