Introduction
Syntactically Controlled Paraphrase Generation (SCPG) aims to generate a sentence that preserves the semantics of a source sentence while adhering to a target syntactic structure. Despite the success of large-scale pre-trained models, achieving fine-grained structural control remains a challenge. Traditional models often struggle with the hierarchical nature of syntax when treated as flat sequences. This paper introduces DSMT, a dual-stage multi-task pre-training scheme designed to imbue the T5 model with deep structural and syntactic awareness.
The Challenge of Structural Control
SCPG requires a model to navigate three primary hurdles:
- Data Scarcity: High-quality triplets of (source, template, target) are rare. Models often "cheat" during training by using the target's own tree as a template, leading to poor generalization.
- Architecture Complexity: Researchers typically choose between complex tree-based encoders, which are difficult to scale, and simple linearization, which may lose structural nuances.
- Shallow Pre-training: Standard language models like T5 are optimized for sequential text, not the non-sequential, hierarchical dependencies of syntax trees.
Methodology: DSMT-T5
The proposed framework, DSMT-T5, utilizes a two-stage pre-training process to bridge the gap between sequential modeling and hierarchical syntax.
Stage 1: Structure-Aware Pre-Training
This stage focuses on the mathematical topology of trees. By replacing linguistic tags (e.g., NP, VP) with anonymous tokens (e.g., <node_1>), the model is forced to learn structural relationships rather than semantic associations. Tasks include treeposition indexing, tree forming from grammar rules, and predicting tree structures after node deletion.
Stage 2: Syntax-Aware Pre-Training
Once the model understands tree topology, real linguistic labels are introduced. The model performs tasks such as Part-of-Speech (POS) tagging, constituency searching, and completing pruned trees. This aligns the previously learned structural representations with actual natural language syntax.
Data Representation
The model processes syntax through Linearized Constituency Trees (LCTs). This approach allows a standard Transformer architecture to handle hierarchical data by treating the tree as a sequence of tokens that the model now "understands" thanks to its specialized pre-training.
Experimental Results
The authors evaluated DSMT-T5 on the ParaNMT dataset and a newly curated high-quality triplet benchmark (~250k samples).
- Performance Leap: In target-based SCPG, DSMT-T5 achieved a BLEU-4 score of 42.3, a staggering improvement over the previous state-of-the-art of 27.8.
- Structural Accuracy: The model showed marked improvements in Tree Edit Distance (TED), indicating that the generated paraphrases followed the target templates more precisely.
- Few-Shot Robustness: DSMT-T5 demonstrated superior few-shot performance, maintaining high quality even when fine-tuned on only 5% of the training data, whereas vanilla T5 performance plummeted.
Conclusion
The DSMT framework demonstrates that specialized pre-training can enable standard Transformer models to master complex hierarchical structures without requiring custom architectures. By decoupling structural learning from linguistic labeling, the model achieves a more robust understanding of syntax, leading to state-of-the-art performance in controlled paraphrase generation. Future work will explore the scalability of these tasks to larger language models to enhance their structural reasoning capabilities.
Summary updated