TL;DR
This paper investigates the reliability of automatic summarization metrics in the challenging domain of call center conversations. By introducing domain-specific "perturbed" summaries, the authors reveal significant brittleness in existing metrics, showing they often fail to align with human judgments. They then demonstrate how these perturbations can be used to train more robust, domain-aware evaluation metrics.
What Problem Does This Paper Address?
Abstractive summarization, which generates novel sentences to condense text, holds immense promise for handling the vast amount of information in call center interactions. However, evaluating these summaries is tricky. Traditional metrics like ROUGE, while useful for general text, struggle with the unique characteristics of call center data: diverse topics, emotional nuances, informal language, and dynamic contexts. This makes it hard to trust that these metrics accurately reflect the quality of summaries generated for customer service scenarios. The paper aims to bridge this gap by assessing and improving summarization metric reliability specifically for the call center domain.
Methods
The core of this research lies in creating "perturbed summaries." These are modified versions of original summaries designed to test the robustness of evaluation metrics. The authors developed several perturbation types, including:
- Writing Style Conversion: Rewriting summaries with different styles (e.g., shorthand, passive voice) while preserving meaning.
- Speaker/Demographic Changes: Altering speaker roles or introducing false demographic information.
- Noise Addition: Injecting irrelevant sentences.
- Length Modification: Shortening or lengthening summaries.
- Category Changes: Shifting the domain or category of the summary.
- Entity-Based Perturbation: Systematically replacing entities (like names or locations) within summaries.
- Sentence-Based Perturbation: Removing key sentences deemed "important" or "non-important" based on their relevance to the call's category.
These perturbations were generated using Large Language Models (LLMs). The authors then used these perturbed summaries to:
- Assess Robustness: Evaluate how existing metrics (like BLEU, ROUGE, BERTScore, UniEval, BARTScore) perform when faced with these modified summaries, comparing their scores to human annotations.
- Augment Training Data: Use the perturbed summaries as additional training data to fine-tune existing metrics or train custom classifier-based metrics, aiming to improve their reliability.
Datasets
The study utilized a proprietary call center dataset comprising 1200 calls across various domains (medical, educational, banking, service). Ground truth summaries were created by seven in-house annotators after a two-week training period. This dataset, along with its perturbed versions, formed the basis for evaluation. The authors also incorporated perturbations from a previous study (Sai et al., 2021) as a "baseline" comparison.
Key Insights and Ideas
- Existing Metrics are Brittle: Off-the-shelf summarization metrics show significant performance degradation when applied to perturbed call center summaries, often correlating poorly with human judgments. This highlights a critical trust issue in using these metrics for this domain.
- Domain-Specific Perturbations Matter: The authors' custom, domain-specific perturbations reveal metric weaknesses more effectively than general perturbations.
- Perturbations as Data Augmentation: Incorporating these perturbed summaries into the training of evaluation metrics (either custom classifiers or fine-tuning existing LLM-based metrics like UniEval and BARTScore) substantially improves their robustness and correlation with human scores.
- LLMs as Perturbation Generators: LLMs are powerful tools for generating diverse and realistic perturbations, enabling a more thorough assessment of metric reliability.
Key Results
- Correlation Drop: For 15 out of 24 metrics, correlation with human scores decreased when evaluated on perturbed data compared to the original data.
- Classifier Performance: When training classifiers on original data and testing on perturbed data, accuracy dropped significantly (e.g., by up to 20.34% for consistency and 20.34% for relevance in some cases). However, when trained on a mix of original and "our" perturbed data, this gap was minimized, showing improved robustness.
- Fine-tuned Metrics Improve: Fine-tuning UniEval and BARTScore with the authors' perturbed data led to enhanced correlation with human consistency and relevance scores, with improvements of up to 8.29% for BARTScore on consistency.
Limitations and Future Work
The authors acknowledge several limitations:
- Perturbation Scope: While multiple perturbations were explored, additional types could further enhance the analysis. The applicability of these perturbations might also vary across different domains.
- LLM Variability: Future LLM versions might produce perturbations of different quality or struggle with prompt consistency.
- Emerging Evaluation Methods: The study did not extensively cover newer LLM-as-a-judge evaluation methods, which would be valuable to explore on perturbed datasets.
Future work could involve exploring a wider range of perturbations, testing their applicability across diverse domains, and evaluating newer evaluation paradigms like LLM-based judges.
Summary updated