TL;DR
This paper introduces a novel kernel-based automatic evaluation method for natural language generation (NLG) tasks, specifically applied to automatic summarization. By leveraging Extended String Subsequence Kernels (ESK) that consider word sequences and even word senses, their method achieves higher correlation with human judgments and greater robustness compared to existing metrics like ROUGE.
What Problem Does This Paper Address?
Evaluating the quality of automatically generated text, such as summaries or translations, is crucial for advancing NLG research. Current automatic evaluation methods often fall short of human judgment, requiring significant manual effort and cost. While existing metrics like ROUGE (based on N-grams or longest common subsequences) have been widely adopted, they can be overly simplistic, failing to capture the nuanced structural similarities between texts, and can be susceptible to "cheating" by generating superficially plausible but semantically weak outputs. This paper aims to bridge this gap by proposing a more sophisticated evaluation metric that better reflects human perception of text quality.
Methods
The core of this paper is the Extended String Subsequence Kernel (ESK), a type of convolution kernel. ESK extends traditional string kernels by not only considering the sequence of words but also allowing for "skips" (i.e., non-contiguous word sequences) and incorporating word senses. This allows the kernel to measure similarities between texts at a deeper structural level, accounting for paraphrasing and semantic equivalence.
The ESK is then used to define a similarity measure between two texts (a system-generated summary and a human reference). This similarity measure is normalized to produce a score between 0 and 1. For automatic evaluation, they define precision-oriented (Pesk) and recall-oriented (Resk) scoring functions, and combine them into a unified F-measure (Fesk). The ESK's ability to handle variable-length skips is controlled by a decay parameter ($\lambda$), and the relative importance of precision and recall in the F-measure is tuned by a parameter ($\beta$).
Datasets
The primary dataset used for evaluation is the Text Summarization Challenge 3 (TSC-3) multiple document summarization dataset. This dataset consists of document sets, for which human-generated reference summaries and system-generated summaries were available. The authors also mention using additional human summaries to examine variations among human assessors.
Key Insights and Ideas
- Beyond N-grams and LCS: The paper demonstrates that simply matching N-grams or the longest common subsequence is insufficient for robust evaluation. ESK's ability to consider longer, non-contiguous word sequences (skip-n-grams) and word senses is key to capturing deeper semantic and structural similarities.
- Word Senses Matter: Incorporating word senses into the similarity calculation allows the metric to be more flexible and robust to paraphrasing, a common challenge in NLG.
- Robustness to "Cheating": The proposed method is less susceptible to "cheating" compared to simpler metrics like ROUGE-1. Metrics that rely solely on high-frequency words or simple matching can be easily gamed, leading to high scores for nonsensical summaries. ESK's structural and semantic considerations make such manipulation more difficult.
- Parameter Sensitivity: While the ESK method shows promise, the authors explore the impact of its parameters ($d$, $\lambda$, $\beta$). They find that the decay parameter for skips ($\lambda$) is significant, and longer skip-n-grams are particularly effective for recall. The optimal parameter settings can vary across datasets.
Key Results
The paper's experimental evaluation on the TSC-3 dataset showed that their ESK-based method (referred to as Fesk) consistently outperformed existing ROUGE metrics, particularly for longer summaries.
- Higher Correlation:
Feskdemonstrated a higher correlation coefficient (r) and Spearman's rank correlation coefficient (ρ) with human judgments compared to ROUGE-N, ROUGE-S, ROUGE-SU, and ROUGE-L. - Robustness: The method was found to be more robust, meaning its performance didn't degrade drastically with minor parameter changes or when stop words were excluded.
- Performance on Long Summaries: The advantage of
Feskover ROUGE was particularly pronounced for longer summaries, where capturing structural coherence is more critical.
Limitations and Future Work
- Parameter Tuning: While the method is robust, optimal parameter settings ($\beta$ and $\lambda$) can vary depending on the task and dataset. The paper suggests that further research is needed to determine optimal parameter values for different tasks.
- Language Independence: The authors acknowledge that to fully prove the language independence of their method, they plan to conduct further experimental evaluations using data from the Document Understanding Conference (DUC).
- Application to Other NLG Tasks: The paper expresses an intention to apply the ESK-based method to other NLG tasks, such as machine translation, to further validate its generalizability.
Summary updated