Which Scores to Predict in Sentence Regression for Text Summarization?

Markus Zopf, Eneldo Loza Mencía, Johannes FürnkranzNAACL 2018 · DOI

Abstract

In extractive text summarization, sentence regression models are typically trained to predict ROUGE recall, aligning with the final evaluation metric. However, this study reveals a counter-intuitive truth: training models to predict ROUGE precision consistently yields superior summaries. By prioritizing information density over raw coverage, greedy selection algorithms can more effectively "pack" relevant content into constrained word budgets. Experiments across multiple English and German benchmarks show that precision-based scoring leads to summaries with significantly higher sentence counts and better overall ROUGE scores, with an optimal performance peak when favoring precision at a weight of approximately 0.9.

Introduction

Extractive summarization involves identifying and concatenating the most salient sentences from a source document. A dominant paradigm is sentence regression: a model assigns a utility score to each sentence, and a greedy algorithm selects the top-scoring candidates until a length constraint (e.g., 100 words) is met. Historically, the "utility" has been defined as ROUGE recall, based on the assumption that maximizing per-sentence recall naturally maximizes the final summary's recall.

The Recall Trap and the Packing Intuition

The authors argue that targeting recall is fundamentally flawed for budget-constrained summarization. High-recall sentences are often long and verbose; they cover many reference concepts but consume significant space. In contrast, ROUGE precision measures information density—the proportion of a sentence that is relevant.

Figure 1: Comparison of sentence selection strategies. Sentence A has high recall but low precision (wordy), while Sentence B has lower individual recall but higher precision (dense).

As illustrated in Figure 1, choosing dense, high-precision sentences allows a greedy algorithm to fit more sentences into the summary. While an individual high-precision sentence might have lower recall than a longer alternative, the aggregate recall of many dense sentences typically surpasses that of a few wordy ones. Summarization is essentially a packing problem where precision acts as the density metric.

Methodology and Datasets

To isolate the impact of the scoring metric, the researchers conducted simulation experiments using both perfect and noisy predictors. They evaluated performance on standard multi-document summarization (MDS) benchmarks, including DUC 2004, TAC 2008, and TAC 2009 for English, alongside the DBS and auto-hMDS corpora for German. The primary variable was the regressand: ROUGE recall vs. ROUGE precision, and their weighted combination $F_\alpha$.

Results and Analysis

The results were consistent across all datasets and languages: ROUGE precision outperformed ROUGE recall. Summaries generated using precision-based scoring contained nearly twice as many sentences as recall-based ones within the same character limits. For instance, on DUC 2004, precision-based summaries averaged ~7.7 sentences compared to ~3.4 for recall-based ones.

Figure 4: ROUGE-1 and ROUGE-2 recall as a function of alpha. Performance peaks across all datasets when alpha is near 0.9, heavily favoring precision.

The analysis of the $F_\alpha$ measure (where $\alpha=0$ is pure recall and $\alpha=1$ is pure precision) showed a clear "sweet spot" at $\alpha \approx 0.9$. As shown in Figure 4, performance peaks when the model heavily favors precision but retains a slight bias toward longer sentences to capture necessary context. This advantage remained robust even when noise was introduced to the predicted scores, suggesting the findings are applicable to real-world, imperfect models.

Conclusion

This research challenges the conventional wisdom of training summarization models to match their evaluation metrics. By shifting the training target from recall to precision, practitioners can leverage the "packing" effect to produce more informative, dense summaries. Future work involves integrating these findings into state-of-the-art neural extractive models by optimizing for precision-based loss functions.

Summary updated

BibTeX
@inproceedings{zopf2018which,
  author = {Markus Zopf and Eneldo Loza Mencía and Johannes Fürnkranz},
  title = {{Which Scores to Predict in Sentence Regression for Text Summarization?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-1161},
}
Figure