Abstract
Sentence Split and Rephrase aims to break down a complex sentence into several simple sentences while preserving its original meaning. Traditional sequence-to-sequence (seq2seq) models often struggle with this task due to two primary limitations: a lack of factual grounding and the inherent order variance of output sentences. This paper introduces Fact-aware Sentence Encoding (FaSE) to improve factual precision and Permutation Invariant Training (PIT) to handle arbitrary sentence ordering. Evaluations on the WebSplit-v1.0 benchmark and extrinsic tests on Open Information Extraction (OpenIE) demonstrate significant performance gains over existing methods.
Introduction
Long and complex sentences are often difficult for both human readers and automated natural language processing (NLP) systems to parse. Tasks such as Open Information Extraction (OpenIE) frequently falter when faced with intricate syntactic structures. The "Sentence Split and Rephrase" task addresses this by decomposing complex inputs into a sequence of shorter, clearer sentences.
However, standard seq2seq models face significant hurdles. First, they often suffer from fact distortion, where the model fails to accurately represent the relationships between entities found in the source text. Second, they are sensitive to order variance. Since there is often no single "correct" order for the resulting simple sentences, standard training objectives—which penalize any deviation from a fixed reference order—can confuse the model and lead to suboptimal performance.
Methodology
To address these challenges, the researchers proposed a framework combining Fact-aware Sentence Encoding (FaSE) and Permutation Invariant Training (PIT).
Fact-aware Sentence Encoding (FaSE)
FaSE utilizes multi-task learning to ensure the encoder captures factual nuances. Alongside the primary generation task, an auxiliary "fact check" classifier is introduced. This classifier is trained to determine if a given fact triple ($arg1, relation, arg2$) is supported by the source sentence. By forcing the encoder to distinguish between true facts and "corrupted" negatives (e.g., where entities or relations are swapped), the model develops internal representations that are much more robust to factual errors.
Permutation Invariant Training (PIT)
To mitigate the issues caused by arbitrary sentence ordering, the authors adapted Permutation Invariant Training from the field of speech separation. Instead of calculating loss against a single fixed target sequence, PIT considers all possible permutations of the target simple sentences. The model then backpropagates only the minimum loss among these permutations. This allows the model to learn the content of the split sentences without being unfairly penalized for choosing a valid but non-standard output order.
Experimental Results
The proposed methods were evaluated on several benchmarks, including WebSplit-v1.0 and WikiSplit.
- WebSplit-v1.0: The LSTM-based model incorporating both PIT and FaSE achieved a BLEU score of 37.3, a substantial improvement over the previous state-of-the-art of 30.5.
- Transformer + Pre-training: By scaling the architecture to a Transformer and pre-training on the large-scale WikiSplit dataset, the performance reached a remarkable 71.0 BLEU.
- Human Evaluation: Blind human assessments confirmed the model's superiority, with high marks for grammaticality (4.78/5.0) and meaning preservation (4.4/5.0).
- Extrinsic Evaluation: Using the model as a preprocessing step for OpenIE systems on the
oie-benchmarkled to improved extraction performance, proving the practical utility of high-quality sentence splitting.
Conclusion
By addressing the core weaknesses of factual inaccuracy and order sensitivity, the FaSE+PIT framework sets a new standard for sentence splitting and rephrasing. The results highlight that factual supervision and flexible training objectives are essential for generating reliable, simplified text. Future work may explore more complex "hard negative" generation for fact-checking and optimizing the computational cost of PIT for longer sequences.
Summary updated