Fact-aware Sentence Split and Rephrase with Permutation Invariant Training

Yinuo Guo, Tao Ge, Furu Wei

Abstract

Sentence Split and Rephrase aims to break down a complex sentence into several simple sentences while preserving its original meaning. Traditional sequence-to-sequence (seq2seq) models often struggle with this task due to two primary limitations: a lack of factual grounding and the inherent order variance of output sentences. This paper introduces Fact-aware Sentence Encoding (FaSE) to improve factual precision and Permutation Invariant Training (PIT) to handle arbitrary sentence ordering. Evaluations on the WebSplit-v1.0 benchmark and extrinsic tests on Open Information Extraction (OpenIE) demonstrate significant performance gains over existing methods.

Introduction

Long and complex sentences are often difficult for both human readers and automated natural language processing (NLP) systems to parse. Tasks such as Open Information Extraction (OpenIE) frequently falter when faced with intricate syntactic structures. The "Sentence Split and Rephrase" task addresses this by decomposing complex inputs into a sequence of shorter, clearer sentences.

However, standard seq2seq models face significant hurdles. First, they often suffer from fact distortion, where the model fails to accurately represent the relationships between entities found in the source text. Second, they are sensitive to order variance. Since there is often no single "correct" order for the resulting simple sentences, standard training objectives—which penalize any deviation from a fixed reference order—can confuse the model and lead to suboptimal performance.

Examples of misrepresented facts and sentence order variance in standard seq2seq models.

Methodology

To address these challenges, the researchers proposed a framework combining Fact-aware Sentence Encoding (FaSE) and Permutation Invariant Training (PIT).

Fact-aware Sentence Encoding (FaSE)

FaSE utilizes multi-task learning to ensure the encoder captures factual nuances. Alongside the primary generation task, an auxiliary "fact check" classifier is introduced. This classifier is trained to determine if a given fact triple ($arg1, relation, arg2$) is supported by the source sentence. By forcing the encoder to distinguish between true facts and "corrupted" negatives (e.g., where entities or relations are swapped), the model develops internal representations that are much more robust to factual errors.

Permutation Invariant Training (PIT)

To mitigate the issues caused by arbitrary sentence ordering, the authors adapted Permutation Invariant Training from the field of speech separation. Instead of calculating loss against a single fixed target sequence, PIT considers all possible permutations of the target simple sentences. The model then backpropagates only the minimum loss among these permutations. This allows the model to learn the content of the split sentences without being unfairly penalized for choosing a valid but non-standard output order.

The proposed architecture featuring the Sentence Encoder with FaSE and the PIT-based loss calculation.

Experimental Results

The proposed methods were evaluated on several benchmarks, including WebSplit-v1.0 and WikiSplit.

  • WebSplit-v1.0: The LSTM-based model incorporating both PIT and FaSE achieved a BLEU score of 37.3, a substantial improvement over the previous state-of-the-art of 30.5.
  • Transformer + Pre-training: By scaling the architecture to a Transformer and pre-training on the large-scale WikiSplit dataset, the performance reached a remarkable 71.0 BLEU.
  • Human Evaluation: Blind human assessments confirmed the model's superiority, with high marks for grammaticality (4.78/5.0) and meaning preservation (4.4/5.0).
  • Extrinsic Evaluation: Using the model as a preprocessing step for OpenIE systems on the oie-benchmark led to improved extraction performance, proving the practical utility of high-quality sentence splitting.

Conclusion

By addressing the core weaknesses of factual inaccuracy and order sensitivity, the FaSE+PIT framework sets a new standard for sentence splitting and rephrasing. The results highlight that factual supervision and flexible training objectives are essential for generating reliable, simplified text. Future work may explore more complex "hard negative" generation for fact-checking and optimizing the computational cost of PIT for longer sequences.

Summary updated

Abstract

Sentence Split and Rephrase aims to break down a complex sentence into several simple sentences with its meaning preserved. Previous studies tend to address the issue by seq2seq learning from parallel sentence pairs, which takes a complex sentence as input and sequentially generates a series of simple sentences. However, the conventional seq2seq learning has two limitations for this task: (1) it does not take into account the facts stated in the long sentence; As a result, the generated simple sentences may miss or inaccurately state the facts in the original sentence. (2) The order variance of the simple sentences to be generated may confuse the seq2seq model during training because the simple sentences derived from the long source sentence could be in any order. To overcome the challenges, we first propose the Fact-aware Sentence Encoding, which enables the model to learn facts from the long sentence and thus improves the precision of sentence split; then we introduce Permutation Invariant Training to alleviate the effects of order variance in seq2seq learning for this task. Experiments on the WebSplit-v1.0 benchmark dataset show that our approaches can largely improve the performance over the previous seq2seq learning approaches. Moreover, an extrinsic evaluation on oie-benchmark verifies the effectiveness of our approaches by an observation that splitting long sentences with our state-of-the-art model as preprocessing is helpful for improving OpenIE performance.
BibTeX
@inproceedings{guo2020fact,
  author = {Yinuo Guo and Tao Ge and Furu Wei},
  title = {{Fact-aware Sentence Split and Rephrase with Permutation Invariant Training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=NP8aeL50V_},
}
Figure