Dual-Stage Multi-Task Syntax-Oriented Pre-Training for Syntactically Controlled Paraphrase Generation

Hongxu Liu, Xiaojie Wang, Jiashen Sun, Ke Zeng, Guanglu Wan

Introduction

Syntactically Controlled Paraphrase Generation (SCPG) aims to generate a sentence that preserves the semantics of a source sentence while adhering to a target syntactic structure. Despite the success of large-scale pre-trained models, achieving fine-grained structural control remains a challenge. Traditional models often struggle with the hierarchical nature of syntax when treated as flat sequences. This paper introduces DSMT, a dual-stage multi-task pre-training scheme designed to imbue the T5 model with deep structural and syntactic awareness.

The Challenge of Structural Control

SCPG requires a model to navigate three primary hurdles:

  1. Data Scarcity: High-quality triplets of (source, template, target) are rare. Models often "cheat" during training by using the target's own tree as a template, leading to poor generalization.
  2. Architecture Complexity: Researchers typically choose between complex tree-based encoders, which are difficult to scale, and simple linearization, which may lose structural nuances.
  3. Shallow Pre-training: Standard language models like T5 are optimized for sequential text, not the non-sequential, hierarchical dependencies of syntax trees.

Methodology: DSMT-T5

The proposed framework, DSMT-T5, utilizes a two-stage pre-training process to bridge the gap between sequential modeling and hierarchical syntax.

Overview of the DSMT-T5 framework, illustrating the transition from structure-aware pre-training (Stage 1) to syntax-aware pre-training (Stage 2) and final SCPG fine-tuning.

Stage 1: Structure-Aware Pre-Training

This stage focuses on the mathematical topology of trees. By replacing linguistic tags (e.g., NP, VP) with anonymous tokens (e.g., <node_1>), the model is forced to learn structural relationships rather than semantic associations. Tasks include treeposition indexing, tree forming from grammar rules, and predicting tree structures after node deletion.

Stage 2: Syntax-Aware Pre-Training

Once the model understands tree topology, real linguistic labels are introduced. The model performs tasks such as Part-of-Speech (POS) tagging, constituency searching, and completing pruned trees. This aligns the previously learned structural representations with actual natural language syntax.

Data Representation

The model processes syntax through Linearized Constituency Trees (LCTs). This approach allows a standard Transformer architecture to handle hierarchical data by treating the tree as a sequence of tokens that the model now "understands" thanks to its specialized pre-training.

Example of a constituency tree and its corresponding linearized sequence used for model input and output.

Experimental Results

The authors evaluated DSMT-T5 on the ParaNMT dataset and a newly curated high-quality triplet benchmark (~250k samples).

  • Performance Leap: In target-based SCPG, DSMT-T5 achieved a BLEU-4 score of 42.3, a staggering improvement over the previous state-of-the-art of 27.8.
  • Structural Accuracy: The model showed marked improvements in Tree Edit Distance (TED), indicating that the generated paraphrases followed the target templates more precisely.
  • Few-Shot Robustness: DSMT-T5 demonstrated superior few-shot performance, maintaining high quality even when fine-tuned on only 5% of the training data, whereas vanilla T5 performance plummeted.

Conclusion

The DSMT framework demonstrates that specialized pre-training can enable standard Transformer models to master complex hierarchical structures without requiring custom architectures. By decoupling structural learning from linguistic labeling, the model achieves a more robust understanding of syntax, leading to state-of-the-art performance in controlled paraphrase generation. Future work will explore the scalability of these tasks to larger language models to enhance their structural reasoning capabilities.

Summary updated

Abstract

Syntactically Controlled Paraphrase Generation (SCPG), which aims at generating sentences having syntactic structures resembling given exemplars, is attracting more research efforts in recent years. We took an empirical survey on previous SCPG datasets and methods and found three tacitly approved while seldom mentioned intrinsic shortcomings/trade-offs in terms of data obtaining, task formulation, and pre-training strategies. As a mitigation to these shortcomings, we proposed a novel Dual-Stage Multi-Task (DSMT) pre-training scheme, involving a series of structure-oriented and syntax-oriented tasks, which, in our opinion, gives sequential text models the ability of com-prehending intrinsically non-sequential structures like Linearized Constituency Trees (LCTs), understanding the underlying syntactics, and even generating them by parsing sentences. We performed further pre-training of the popular T5 model on these novel tasks and fine-tuned the trained model on every possible variant of SCPG task in literature, finding that our models significantly outperformed (up to 10+ BLEU-4) previous state-of-the-art methods. Finally, we carried out ablation studies which demonstrated the effectiveness of our DSMT methods and emphasized on the SCPG performance gains compared to vanilla T5 models, especially on hard samples or under few-shot settings.
BibTeX
@inproceedings{liu2024dual,
  author = {Hongxu Liu and Xiaojie Wang and Jiashen Sun and Ke Zeng and Guanglu Wan},
  title = {{Dual-Stage Multi-Task Syntax-Oriented Pre-Training for Syntactically Controlled Paraphrase Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.845},
}
Figure