Retrofitting Structure-aware Transformer Language Model for End Tasks

Hao Fei, Yafeng Ren, ji Donghong

Abstract: We consider retrofitting structure-aware Transformer-based language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.

Introduction

While large-scale Transformer models like BERT implicitly capture some linguistic structure, they often struggle with tasks that rely heavily on precise syntactic knowledge. Existing "Tree Transformers" typically suffer from two main issues: "Full-stack Interference," where structural constraints across all layers can degrade high-level semantic performance, and "Structural Monoculture," focusing on either constituency or dependency exclusively. This paper introduces a method to "retrofit" Transformers with explicit syntactic knowledge using a middle-layer structural learning module, preserving the semantic richness of the top layers while enhancing linguistic awareness.

Methods

The authors propose a Structure-aware Transformer that integrates a standalone learning module into the middle of a standard Transformer stack. This approach is built on three pillars:

  1. Syntactic Distance: Instead of complex tree-traversal, the model uses "syntactic distance"—a scalar value for each word representing its height in a parse tree. This single metric simultaneously represents phrase boundaries (constituency) and head-dependent relationships (dependency).
  2. Middle-Layer Injection: Based on probing experiments, the authors identified layer 6 (in a 12-layer model) as the "sweet spot" for syntax. The structural learning is injected here using a weighted sum of representations from layers 5, 6, and 7.
  3. Structure-aware Fine-tuning: Unlike previous models that freeze their structure after pre-training, this model allows the structure module to be fine-tuned alongside the end task, letting the model adjust its internal trees to the specific granularity required by the task.
The Structure-aware Transformer architecture featuring the Structure Learning Module integrated via a weighted sum of middle-layer representations.

Key Insights and Results

The most significant insight is the Middle-Layer Hypothesis: syntax acts as a bridge, not a destination. By confining structural learning to the middle layers, the model preserves the lower layers for feature extraction and the upper layers for task-specific reasoning.

Probing results for constituency phrase parsing and dependency alignment across different layers, showing a clear performance peak at layer 6.

Experimental results validate this approach across several dimensions:

  • Language Modeling: The retrofitted model achieved a perplexity of 16.2, significantly lower than the vanilla BERT baseline of 21.5.
  • Parsing: In unsupervised constituency parsing, the model achieved a 60.3% F1 score, outperforming specialized models like PRPN (42.8%) and DIORA (56.2%).
  • Downstream Performance: On a suite of 8 NLP tasks (including NER, Sentiment Analysis, and Semantic Role Labeling), the structure-aware BERT outperformed the standard baseline by an average of 3.52 points.
  • Task-Specific Granularity: Different tasks require different tree structures. For example, Sentiment Analysis (SST) benefits from coarser phrases, while Semantic Role Labeling (SRL) requires finer-grained phrase lengths. The fine-tuning approach allows the model to adapt accordingly.

Conclusion

Retrofitting Transformers with a middle-layer structure module provides a modular and effective way to upgrade existing architectures. The findings suggest that "perfect" linguistic trees aren't always the most useful for every task; unsupervised induction often performs better for semantic-heavy tasks than supervised injection. Future work will explore scaling this retrofitting to larger models where implicit structure might already be more robust.

Summary updated

Abstract

We consider retrofitting structure-aware Transformer language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.
BibTeX
@inproceedings{fei2020retrofitting,
  author = {Hao Fei and Yafeng Ren and ji Donghong},
  title = {{Retrofitting Structure-aware Transformer Language Model for End Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=r9_441mzpS8},
}
Figure