Abstract
This paper presents a semi-automated methodology for rapidly building a high-quality sentence fusion corpus using Amazon Mechanical Turk. By implementing a novel "representative response" selection technique to aggregate crowdworker outputs, the authors achieved 95% accuracy for union fusion, providing a scalable path for training supervised models in text summarization.
Introduction
Sentence fusion—the process of merging parts of multiple sentences into a single, coherent one—is a critical component for advanced summarization and question-answering systems. While extractive summarization (simply picking existing sentences) is common, it often includes irrelevant information or awkward phrasing. The bottleneck for improving automated fusion is the lack of large-scale, human-annotated datasets. Creating these manually is prohibitively slow and expensive. While crowdsourcing platforms like Amazon Mechanical Turk (AMT) offer a solution, they excel at simple labeling. Sentence fusion requires "free text" entry, which leads to high variance and inconsistent interpretations among workers, making it difficult to produce a "gold standard" corpus.
Methodology
The researchers developed a three-stage pipeline to create a reliable corpus efficiently:
1. Semi-Automated Pair Selection
To find sentences worth fusing, data was pulled from news clusters generated by the Newsblaster summarization system. Various similarity metrics were tested, and bigram overlap was found to be the most effective at identifying pairs that were neither too different to fuse nor already identical. A "sweet spot" threshold between 0.35 and 0.65 was established.
2. Crowdsourced Annotation
AMT workers were tasked with two distinct types of fusion:
- Intersection Fusion: A sentence containing only the information common to both inputs.
- Union Fusion: A sentence containing all information from both inputs.
3. Representative Response Selection
To handle the diversity of crowd responses, five versions were collected for every pair. Instead of picking one at random, a similarity-based algorithm was used. By calculating the cosine similarity (weighted by tf-idf using the Gigaword corpus) between all pairs of responses for a single task, the sentence with the highest overlap with its peers was selected. This "consensus" approach filters out idiosyncratic or low-quality responses.
Key Results
The representative selection method significantly improved the quality of the final corpus compared to the average individual response.
As shown in the results, union fusion achieved an impressive 95% accuracy using the representative selection method. However, intersection fusion proved more challenging, reaching only 54% accuracy.
Error Analysis
The disparity between union and intersection performance prompted a detailed error analysis to understand worker behavior.
The primary reason for the lower intersection score was "task confusion." Workers frequently provided a union (all info) when asked for an intersection (common info). As seen in the error breakdown, "Union/Intersection" errors were the most frequent for intersection tasks (46 cases). When this specific error was excluded, intersection accuracy jumped by 22%.
Conclusion
This research demonstrates that for generative text tasks, the "wisdom of the crowd" can be captured by selecting the most representative sentence. This methodology is remarkably fast: the entire 300-pair experiment was completed in just ten hours of expert labor and seven days of crowd activity. While intersection fusion requires more refined prompting to avoid task confusion, the high accuracy achieved for union fusion provides a scalable path for creating the large-scale datasets necessary for production-grade automated sentence fusion.
Summary updated