A Study of Re-generating Sentences Given Similar Sentences that Cover Them on the Level of Form and Meaning

Hsuan-Wei Lo, Yifei Zhou, Rashel Fam, Yves LepagePACLIC 2022

Abstract

This research introduces the novel NLP task of sentence re-generation, where a system must reconstruct a target sentence using fragments extracted from similar sentences. We evaluate two distinct methodologies—a naive permutation-based language model and a neural Transformer-based approach—across three diverse datasets: Multi30K, Tatoeba, and ACL ARC. Our findings demonstrate that the neural approach significantly outperforms traditional reordering methods in capturing both the literal form and semantic meaning of the target, particularly in complex domains.

Introduction

In Machine Translation, "Translation Memories" (TM) serve as repositories of previously translated segments. While Neural Machine Translation (NMT) systems can utilize these as "hints" to improve output quality, the field lacks a standardized benchmark to evaluate how effectively a system can synthesize these retrieved fragments into a coherent target sentence. This paper addresses this gap by formalizing the sentence re-generation task and providing over 4 million annotated sentences for benchmarking.

The Sentence Re-generation Task

The core challenge of sentence re-generation is to "solve the puzzle" of a sentence by assembling pieces found in other similar sentences. The authors define two primary dimensions for this task:

  1. Form: Reconstructing the target using literal substrings found in retrieved sentences.
  2. Meaning: Reconstructing the target using semantically similar fragments that may not be exact word-for-word matches.
Figure 1: Examples of Form and Meaning coverage for the input sentence "two young toddlers outside on the grass."

Methodology

To evaluate the task, the researchers compared two distinct philosophies:

  • Naive Baseline (Permutation + LM): This approach identifies overlapping fragments (covering parts), generates all possible permutations, and utilizes a language model (KenLM) to select the sequence with the lowest perplexity.
  • Neural Approach (NMT-style): This method treats re-generation as a sequence-to-sequence translation task. Concatenated fragments are fed into a Transformer model (via OpenNMT) trained to "translate" disjointed pieces back into the original, coherent sentence. Sub-word tokenization via SentencePiece is employed to handle rare vocabulary and technical terms.

Datasets and Complexity

Three language resources were developed by annotating existing corpora with XML-like tags to identify overlapping segments:

  • Multi30K: Simple image captions, providing a baseline for short, descriptive text.
  • Tatoeba: A diverse set of short sentences from a language-learning community.
  • ACL ARC: Academic articles in computational linguistics, representing the most difficult challenge due to long sentences and specialized terminology.

Key Results and Analysis

The neural approach demonstrated a clear superiority in reconstructing sentences, particularly when evaluated on the level of "Form."

Figure 6: Performance comparison between Naive and Neural approaches across three corpora, measuring Accuracy, Edit Distance, and BLEU scores.

Form-Level Performance

As shown in Figure 6, the neural model achieved significantly higher exact-match accuracy and lower edit distances compared to the naive baseline. For instance, on the Tatoeba dataset, the neural model reached an accuracy of 24.64%, whereas the naive baseline struggled at 0.44%. The neural model also reduced the character-level edit distance by nearly 42% on the Multi30K dataset.

Meaning-Level Performance

On the level of "Meaning," the neural approach consistently outperformed the baseline, achieving an average BERTScore of 0.89 to 0.91. While the naive baseline occasionally scored higher in BLEU points (as seen in Multi30K and ACL ARC), this is attributed to the baseline being forced to use exact retrieved strings, whereas the neural model often paraphrases or simplifies text to improve overall coherence and semantic accuracy.

Conclusion

This study establishes sentence re-generation as a viable and challenging NLP task. While neural models show great promise in synthesizing fragments, significant hurdles remain, particularly in low-coverage scenarios like the ACL ARC dataset (where only 63% of the target form is typically covered). Future work will focus on refining semantic coverage metrics and expanding the task to multilingual contexts to test cross-linguistic generalizability.

Summary updated

BibTeX
@inproceedings{lo2022study,
  author = {Hsuan-Wei Lo and Yifei Zhou and Rashel Fam and Yves Lepage},
  title = {{A Study of Re-generating Sentences Given Similar Sentences that Cover Them on the Level of Form and Meaning}},
  booktitle = {PACLIC},
  year = {2022},
}
Figure