Abstract
Sentence Split and Rephrase (SSR) aims to decompose complex sentences into multiple simple sentences while preserving the original meaning. Traditional sequence-to-sequence (seq2seq) models often suffer from fact distortion and order variance, where the model either misrepresents factual content or is penalized for valid but non-canonical sentence orderings. This paper introduces two novel components: Fact-aware Sentence Encoding (FaSE) and Permutation Invariant Training (PIT). FaSE uses multi-task learning to ground the encoder in factual triples, while PIT allows the model to optimize for the best-matching permutation of target sentences. Experiments on the WebSplit-v1.0 benchmark and extrinsic evaluations on Open Information Extraction (OpenIE) demonstrate that these approaches significantly outperform existing baselines.
Introduction
Sentence Split and Rephrase (SSR) is a vital preprocessing task in Natural Language Processing, particularly for downstream applications like Machine Translation and Information Extraction that struggle with long-tail linguistic complexity. By breaking a long, convoluted sentence into several shorter, simpler ones, SSR can significantly improve the performance of these systems. However, current seq2seq models face two major hurdles: they often "hallucinate" or omit key facts, and they struggle with the inherent label ambiguity caused by the arbitrary order of independent simple sentences.
Methodology
To overcome these challenges, the authors propose an enhanced encoder-decoder architecture that explicitly models factual content and handles order variance.
Fact-Aware Sentence Encoding (FaSE)
FaSE addresses fact distortion by incorporating a multi-task learning objective. During training, the sentence encoder is tasked not only with generating the simplified sentences but also with a classification task: determining whether a given factual triple (subject-relation-object) is supported by the source sentence. This forces the encoder to produce "fact-aware" representations ($h_s$), ensuring the decoder has access to grounded factual information.
Permutation Invariant Training (PIT)
To resolve the order variance problem, the authors adapt Permutation Invariant Training (PIT) from the field of speech separation. In standard training, a model is penalized if its output order does not exactly match the reference, even if the content is correct. PIT instead considers all possible permutations of the target simple sentences and backpropagates only the minimum loss. This allows the model to converge more effectively by focusing on content accuracy rather than arbitrary sequencing.
Experimental Results
The proposed model achieved a new state-of-the-art on the WebSplit-v1.0 benchmark, raising the BLEU score from 30.5 to 37.3. When pre-trained on the larger WikiSplit dataset using a Transformer architecture, the performance reached 71.0 BLEU, a significant improvement over previous methods. Human evaluations further confirmed these results, with the model scoring 3.62/5 on a composite metric of grammar and simplicity, nearing the human reference score of 3.74.
Extrinsic Evaluation: OpenIE
The utility of the SSR model was further validated through an extrinsic evaluation on the Stanford OpenIE system. By using the SSR model as a pre-processor to simplify complex sentences before extraction, the system's precision-recall curve shifted significantly upward.
As shown in the figure, the "Our_Stanford" approach consistently outperforms baseline configurations across both in-domain and benchmark datasets, proving that effective sentence splitting is a powerful "force multiplier" for information extraction tasks.
Conclusion
This paper demonstrates that grounding seq2seq models in factual reality and accounting for output order flexibility are key to mastering the Sentence Split and Rephrase task. The combination of FaSE and PIT provides a robust framework that not only achieves state-of-the-art intrinsic performance but also delivers measurable improvements for downstream NLP pipelines. Future work will explore the scalability of PIT for tasks requiring a larger number of split fragments.
Summary updated