Abstract
This research introduces the novel NLP task of sentence re-generation, where a system must reconstruct a target sentence using fragments extracted from similar sentences. We evaluate two distinct methodologies—a naive permutation-based language model and a neural Transformer-based approach—across three diverse datasets: Multi30K, Tatoeba, and ACL ARC. Our findings demonstrate that the neural approach significantly outperforms traditional reordering methods in capturing both the literal form and semantic meaning of the target, particularly in complex domains.
Introduction
In Machine Translation, "Translation Memories" (TM) serve as repositories of previously translated segments. While Neural Machine Translation (NMT) systems can utilize these as "hints" to improve output quality, the field lacks a standardized benchmark to evaluate how effectively a system can synthesize these retrieved fragments into a coherent target sentence. This paper addresses this gap by formalizing the sentence re-generation task and providing over 4 million annotated sentences for benchmarking.
The Sentence Re-generation Task
The core challenge of sentence re-generation is to "solve the puzzle" of a sentence by assembling pieces found in other similar sentences. The authors define two primary dimensions for this task:
- Form: Reconstructing the target using literal substrings found in retrieved sentences.
- Meaning: Reconstructing the target using semantically similar fragments that may not be exact word-for-word matches.
Methodology
To evaluate the task, the researchers compared two distinct philosophies:
- Naive Baseline (Permutation + LM): This approach identifies overlapping fragments (covering parts), generates all possible permutations, and utilizes a language model (KenLM) to select the sequence with the lowest perplexity.
- Neural Approach (NMT-style): This method treats re-generation as a sequence-to-sequence translation task. Concatenated fragments are fed into a Transformer model (via OpenNMT) trained to "translate" disjointed pieces back into the original, coherent sentence. Sub-word tokenization via SentencePiece is employed to handle rare vocabulary and technical terms.
Datasets and Complexity
Three language resources were developed by annotating existing corpora with XML-like tags to identify overlapping segments:
- Multi30K: Simple image captions, providing a baseline for short, descriptive text.
- Tatoeba: A diverse set of short sentences from a language-learning community.
- ACL ARC: Academic articles in computational linguistics, representing the most difficult challenge due to long sentences and specialized terminology.
Key Results and Analysis
The neural approach demonstrated a clear superiority in reconstructing sentences, particularly when evaluated on the level of "Form."
Form-Level Performance
As shown in Figure 6, the neural model achieved significantly higher exact-match accuracy and lower edit distances compared to the naive baseline. For instance, on the Tatoeba dataset, the neural model reached an accuracy of 24.64%, whereas the naive baseline struggled at 0.44%. The neural model also reduced the character-level edit distance by nearly 42% on the Multi30K dataset.
Meaning-Level Performance
On the level of "Meaning," the neural approach consistently outperformed the baseline, achieving an average BERTScore of 0.89 to 0.91. While the naive baseline occasionally scored higher in BLEU points (as seen in Multi30K and ACL ARC), this is attributed to the baseline being forced to use exact retrieved strings, whereas the neural model often paraphrases or simplifies text to improve overall coherence and semantic accuracy.
Conclusion
This study establishes sentence re-generation as a viable and challenging NLP task. While neural models show great promise in synthesizing fragments, significant hurdles remain, particularly in low-coverage scenarios like the ACL ARC dataset (where only 63% of the target form is typically covered). Future work will focus on refining semantic coverage metrics and expanding the task to multilingual contexts to test cross-linguistic generalizability.
Summary updated