Power mean pyramid scores for summarization evaluation

Sameer Maskey, Andrew RosenbergInterspeech 2012

TL;DR

This paper introduces Power Mean Pyramid Scores (PMP), a novel evaluation metric for automatic text summarization that improves upon the standard Pyramid method. By using a power mean to combine the scores of individual summary content units (SCUs), PMP can be tuned to achieve significantly higher correlations with human judgments of summary quality, offering a more robust and adaptable evaluation framework.

What Problem Does This Paper Address?

Evaluating the quality of automatically generated text summaries is a significant challenge. While human-generated summaries serve as a gold standard, comparing machine summaries to them requires reliable metrics. The Pyramid method is a popular approach, assessing summaries based on how many reference summaries contain specific "summary content units" (SCUs) – atomic facts independent of their wording. However, the standard Pyramid method uses a simple linear combination of SCU scores, which might not optimally capture human perception of quality. This paper aims to address this limitation by proposing a more flexible and tunable combination function for SCU scores within the Pyramid framework.

Methods

The core innovation of this paper is the introduction of the Power Mean Pyramid Scoring (PMP) method. PMP builds upon the existing Pyramid scoring scheme. Here's how it works:

  1. SCU Scoring (Standard Pyramid): Like the original Pyramid method, PMP first identifies Summary Content Units (SCUs) from reference summaries. Each SCU is then assigned a weight based on how many reference summaries contain it.
  2. Power Mean Combination: Instead of a simple linear average, PMP uses the power mean to combine the weights of the SCUs present in a system-generated summary. The power mean is a generalization of arithmetic, geometric, and harmonic means, parameterized by an exponent 'α'.
  3. The formula for the power mean is: $M(\alpha, x_1, ..., x_n) = (\frac{1}{n} \sum_{i=1}^{n} x_i^\alpha)^{\frac{1}{\alpha}}$
    • By varying 'α', the power mean can behave like a minimum (precision-biased) or maximum (recall-biased) function, or anything in between.
  4. Normalization: PMP normalizes the resulting power mean score to ensure it ranges from zero to one, similar to the original Pyramid method.

The key idea is that by tuning the 'α' parameter, PMP can find a combination function that best aligns with human judgments of summary quality for a specific task, dataset, and annotation style.

Datasets

The evaluation was performed using the TAC-08 Opinion Summarization shared task evaluation material. This dataset consists of opinion questions where systems were required to summarize justifications for a particular point of view. The data included:

  • 22 targets, each with multiple sub-questions.
  • Systems generated summaries for each sub-question.
  • Human annotators created 10 opinion summaries for each target to serve as reference summaries.
  • Both reference and system summaries were manually annotated for SCUs.
  • Human judgments were collected on a scale of 1 to 10 for Overall Responsiveness Judgement (ORJ).

The data was split into tuning and test sets, with 11 targets (382 summaries) for tuning and 11 targets (378 summaries) for testing.

Key Insights and Ideas

  • Flexibility is Key: The standard Pyramid method's linear combination is a "one-size-fits-all" approach. PMP demonstrates that a tunable combination function (the power mean) can significantly improve evaluation metric performance.
  • Tuning Matters: The optimal 'α' value for the power mean is dataset and task-dependent. This suggests that evaluation metrics might need to be adapted rather than universally applied.
  • PMP is Robust: The PMP metric showed consistent performance across both tuning and test sets, indicating its reliability and potential for generalization.
  • Beyond Linear: The paper highlights that simple arithmetic means (like in the standard Pyramid) might not be the best way to combine component scores for summarization quality.

Key Results

  • Improved Correlation: On the TAC-08 test set, PMP achieved significantly higher correlations with human judgments (ORJ) compared to the standard Pyramid Score and the TAC-08 metric.
  • Spearman correlation: PMP (0.2417) vs. Pyramid (0.169) and TAC-08 (0.204).
  • Pearson correlation: PMP (0.2307) vs. Pyramid (0.169) and TAC-08 (0.198).
  • Kendall correlation: PMP (0.1693) vs. Pyramid (0.128) and TAC-08 (0.143).
  • Optimal 'α' Range: The optimal 'α' value for PMP on this dataset was found to be between the geometric mean (α=0) and the arithmetic mean (α=1), suggesting a preference for a combination that balances precision and recall.
  • TAC-08 Metric Sub-optimality: The paper suggests that the TAC-08 metric, while designed for the task, might not have been optimally configured, as PMP showed better correlation even with its tuned parameter.

Limitations and Future Work

  • Tuning Overhead: While PMP offers improved performance, it requires an additional tuning step to find the optimal 'α' value. The paper notes that this tuning is done on a held-out set and the performance is consistent on the test set.
  • Generalizability: The authors suggest that the power mean framework for combining scores could be applied to other evaluation measures like ROUGE and BLEU, which would be a natural direction for future work.
  • Task Specificity: The optimal 'α' was found for a specific task (opinion summarization) and dataset. Further research could explore how 'α' varies across different summarization tasks (e.g., news summarization, query-focused summarization) and datasets.

Summary updated

BibTeX
@inproceedings{maskey2012power,
  author = {Sameer Maskey and Andrew Rosenberg},
  title = {{Power mean pyramid scores for summarization evaluation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure