Time-Efficient Creation of an Accurate Sentence Fusion Corpus

Kathleen R. McKeown, Sara Rosenthal, Kapil Thadani, Coleman MooreNAACL 2010 · OpenReview

Abstract

This paper presents a semi-automated methodology for rapidly building a high-quality sentence fusion corpus using Amazon Mechanical Turk. By implementing a novel "representative response" selection technique to aggregate crowdworker outputs, the authors achieved 95% accuracy for union fusion, providing a scalable path for training supervised models in text summarization.

Introduction

Sentence fusion—the process of merging parts of multiple sentences into a single, coherent one—is a critical component for advanced summarization and question-answering systems. While extractive summarization (simply picking existing sentences) is common, it often includes irrelevant information or awkward phrasing. The bottleneck for improving automated fusion is the lack of large-scale, human-annotated datasets. Creating these manually is prohibitively slow and expensive. While crowdsourcing platforms like Amazon Mechanical Turk (AMT) offer a solution, they excel at simple labeling. Sentence fusion requires "free text" entry, which leads to high variance and inconsistent interpretations among workers, making it difficult to produce a "gold standard" corpus.

Methodology

The researchers developed a three-stage pipeline to create a reliable corpus efficiently:

1. Semi-Automated Pair Selection

To find sentences worth fusing, data was pulled from news clusters generated by the Newsblaster summarization system. Various similarity metrics were tested, and bigram overlap was found to be the most effective at identifying pairs that were neither too different to fuse nor already identical. A "sweet spot" threshold between 0.35 and 0.65 was established.

2. Crowdsourced Annotation

AMT workers were tasked with two distinct types of fusion:

  • Intersection Fusion: A sentence containing only the information common to both inputs.
  • Union Fusion: A sentence containing all information from both inputs.

3. Representative Response Selection

To handle the diversity of crowd responses, five versions were collected for every pair. Instead of picking one at random, a similarity-based algorithm was used. By calculating the cosine similarity (weighted by tf-idf using the Gigaword corpus) between all pairs of responses for a single task, the sentence with the highest overlap with its peers was selected. This "consensus" approach filters out idiosyncratic or low-quality responses.

Key Results

The representative selection method significantly improved the quality of the final corpus compared to the average individual response.

Accuracy of fusion responses across all tasks vs. selected representative responses.

As shown in the results, union fusion achieved an impressive 95% accuracy using the representative selection method. However, intersection fusion proved more challenging, reaching only 54% accuracy.

Error Analysis

The disparity between union and intersection performance prompted a detailed error analysis to understand worker behavior.

Breakdown of error types for intersection and union fusion tasks.

The primary reason for the lower intersection score was "task confusion." Workers frequently provided a union (all info) when asked for an intersection (common info). As seen in the error breakdown, "Union/Intersection" errors were the most frequent for intersection tasks (46 cases). When this specific error was excluded, intersection accuracy jumped by 22%.

Conclusion

This research demonstrates that for generative text tasks, the "wisdom of the crowd" can be captured by selecting the most representative sentence. This methodology is remarkably fast: the entire 300-pair experiment was completed in just ten hours of expert labor and seven days of crowd activity. While intersection fusion requires more refined prompting to avoid task confusion, the high accuracy achieved for union fusion provides a scalable path for creating the large-scale datasets necessary for production-grade automated sentence fusion.

Summary updated

Abstract

Kathleen McKeown, Sara Rosenthal, Kapil Thadani, Coleman Moore. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. 2010.
BibTeX
@inproceedings{mckeown2010time,
  author = {Kathleen R. McKeown and Sara Rosenthal and Kapil Thadani and Coleman Moore},
  title = {{Time-Efficient Creation of an Accurate Sentence Fusion Corpus}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=H1x8y7WOWH},
}
Figure