Abstract
Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks. However, it is unclear what the learned attention captures. The attention computed by attention heads seems not to match human intuitions about hierarchical structures. This paper proposes Tree Transformer, which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures. The tree structures can be automatically induced from raw texts by our proposed "Constituent Attention" module, which is simply implemented by self-attention between two adjacent words. With the same training procedure identical to BERT, the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures, better language modeling, and further learning more explainable attention scores.
Introduction
Standard Transformer architectures have significantly advanced natural language processing by modeling text as a sequence of tokens. However, these models typically treat sequences as "flat," overlooking the inherent hierarchical structure of human language where words form constituents like phrases and clauses. While self-attention allows for global dependencies, the resulting attention patterns often lack linguistic interpretability and do not naturally align with tree-like structures.
The Tree Transformer addresses this by introducing a "Constituent Attention" module. This module constrains the standard self-attention mechanism to follow induced tree structures without requiring human-annotated labels. By integrating these structural priors, the model aims to improve language modeling performance and generate more explainable attention maps.
Methods
The Tree Transformer modifies the bidirectional Transformer encoder through three key mechanisms:
- Constituent Attention and Prior: The model computes a "Constituent Prior" matrix ($C$), representing the probability that two words belong to the same constituent. The standard attention matrix ($E$) is modified via element-wise multiplication with this prior: $E = C \odot \text{softmax}(QK^T / \sqrt{d})$. This effectively masks attention, restricting words to primarily attend to others within their own linguistic group.
- Hierarchical Constraints: To ensure the emergence of a valid tree structure, the model enforces a constraint where constituents at lower layers must be nested within those of higher layers. As information ascends through the network, smaller word groups (e.g., "the cute dog") are merged into larger constituents.
- Neighboring Attention: The process is bootstrapped using a "link" mechanism where each word learns to attend to its immediate neighbors. These local links serve as the basis for calculating constituent probabilities.
The model is trained using the standard Masked Language Modeling (MLM) objective, allowing it to learn hierarchical structures in a completely unsupervised manner.
Experimental Setup and Results
The Tree Transformer was evaluated on the Penn Treebank (PTB) dataset, including the WSJ-10 subset and the full WSJ test set. Performance was compared against unsupervised parsing baselines such as PRPN, On-LSTM, and Compound PCFG.
Language Modeling and Parsing Accuracy
The Tree Transformer demonstrated superior language modeling capabilities, achieving a perplexity of 45.6 (12 layers), compared to 48.1 for a standard Transformer of equivalent size. In terms of unsupervised grammar induction, the model achieved an F1 score of approximately 50.5–52.0% on the WSJ-test set. While slightly below specialized PCFG models, it significantly outperformed standard Transformer baselines.
The model showed high recall for specific linguistic categories, notably Noun Phrases (67.6%) and Adverb Phrases (55.1%), indicating a strong ability to identify core structural components of sentences.
Discussion and Conclusion
A significant advantage of the Tree Transformer is its emergent interpretability. Unlike the often noisy attention maps of standard Transformers, the Tree Transformer produces clear, block-diagonal patterns that correspond to phrases. These blocks expand in higher layers, visually tracing the construction of a parse tree.
However, certain challenges remain. Initializing the model with pre-trained BERT weights proved counterproductive, as BERT's "flat" attention patterns conflict with the strict hierarchical constraints. Additionally, the Constituent Attention module introduces a minor computational overhead, increasing parameter count by approximately 10% and slowing inference by 1.2x. Future work may explore more sophisticated top-down decoding algorithms to further refine the quality of induced trees and reduce reliance on positional encodings.
In summary, the Tree Transformer successfully integrates hierarchical priors into the self-attention mechanism, proving that structural constraints can enhance both the performance and the explainability of large-scale language models.
Summary updated