EmailSum: Abstractive Email Thread Summarization

Shiyue Zhang, Asli Celikyilmaz, Jianfeng Gao, Mohit Bansal

TL;DR

This paper introduces EMAILSUM, the first large-scale dataset for abstractive email thread summarization, and presents a comprehensive study of various summarization models. They found that while pre-trained models like T5 provide a strong baseline, semi-supervised learning can improve performance, but human evaluation reveals significant challenges in understanding sender intent and speaker roles, highlighting the limitations of current automatic evaluation metrics.

What Problem Does This Paper Address?

In today's information-saturated world, efficiently processing lengthy email threads is crucial for productivity. While single-document summarization has been extensively studied, summarizing multi-turn conversations like email threads presents unique challenges due to their structure, length, and the need to capture evolving context and speaker dynamics. Existing research in thread summarization is limited by a lack of large, high-quality datasets, hindering the development and evaluation of robust abstractive summarization models for this domain.

Methods

The authors explore a range of summarization techniques, categorizing them into extractive and abstractive methods.

Extractive Methods: * Oracle: A theoretical upper bound that maximizes an evaluation metric. * Lead: Simple heuristics like picking the first sentence of the thread (Lead-1) or the first sentence of each email (Lead-1-Email). * TextRank: A graph-based method that ranks sentences based on embedding similarity. * BertSumExt: An extractive summarizer leveraging BERT.

Abstractive Methods: * Fast Abs RL: A hybrid model that first extracts sentences and then rewrites them abstractively, adapted to extract emails instead of sentences. * T5: A powerful Transformer-based seq-to-seq model fine-tuned on email threads, serving as a primary baseline (T5base). * Transfer Learning: Investigating the impact of pre-training or joint-training T5 on other summarization datasets (CNN/DM, XSum, SAMSum, CRD3). * Semi-supervised Learning: Utilizing unlabeled email threads to improve model performance, primarily through a self-training approach. * Hierarchical T5: A novel architecture that uses separate encoders for tokens within emails and for the emails themselves, aiming to capture the hierarchical structure of threads.

Datasets

  • EMAILSUM Dataset: A newly introduced dataset containing 2,549 human-annotated email threads with both short (<30 words) and long (<100 words) abstractive summaries. This dataset is significantly larger than previous email thread summarization datasets.
  • Unlabeled Data: An additional 8,594 unlabeled email threads from the Avocado and W3C collections were used for semi-supervised learning.
  • Existing Datasets (for transfer learning): CNN/DM (news), XSum (news), SAMSum (dialogue), CRD3 (dialogue), and BC3 (email thread).

Key Insights and Ideas

  • The Importance of Pre-trained Models: T5, a large pre-trained Transformer, provides a surprisingly strong baseline for email thread summarization, even when treating the entire thread as a single document. This highlights the power of transfer learning from general language understanding.
  • Semi-Supervised Learning's Double-Edged Sword: While semi-supervised learning with unlabeled data significantly boosts automatic metrics like ROUGE, human evaluations suggest these models tend to become more extractive, which humans don't always prefer.
  • Human Judgment is Key (and Hard to Automate): Automatic metrics like ROUGE and BERTScore show weak correlation with human judgments on this task. This underscores the critical need for human evaluation and the development of better automatic metrics that capture nuances like sender intent and role identification.
  • Hierarchical Structure Offers Limited Gains: While intuitively appealing, a hierarchical T5 model only marginally improved performance over the non-hierarchical baseline, suggesting that the standard T5 architecture is already adept at handling the sequential nature of email threads.
  • Understanding Intent and Roles is the Next Frontier: The most significant failures of current models lie in grasping the underlying intent of the sender and correctly identifying the roles and relationships between participants in the thread.

Key Results

  • The T5base model (fine-tuned T5 on email threads) served as a strong baseline.
  • Semi-supervised learning (SemiSup) models achieved the best ROUGE-1 scores, with SemiSuptogether outperforming T5base by a significant margin on EMAILSUMshort and SemiSupw3c on EMAILSUMlong.
  • However, in human evaluations, the best ROUGE-1 models (SemiSup) were often rated lower than the T5base model in terms of overall quality, salience, and faithfulness, indicating a disconnect between automatic metrics and human preference.
  • Human evaluators rated model-generated summaries highly (around 3.89-4.29 out of 5) for salience and faithfulness, but identified specific error patterns: failing to understand sender intent and failing to identify speaker roles.
  • Performance (ROUGE-1) generally decreased as the number of emails in a thread increased.

Limitations and Future Work

  • Limitations of Automatic Metrics: The paper strongly emphasizes that ROUGE and BERTScore are poorly correlated with human judgments for email thread summarization, necessitating the development of new, more nuanced evaluation metrics.
  • Model Failures: Current models struggle with understanding the sender's high-level intent and identifying the roles of different speakers, leading to summaries that may be factually correct but miss the core message or misattribute information.
  • Future Work: The authors call for the community to develop better metrics that can capture these crucial aspects of summarization quality. They also suggest further research into hierarchical models and more sophisticated semi-supervised or transfer learning techniques that can better address the specific challenges of email thread summarization.

Summary updated

BibTeX
@inproceedings{zhang2021emailsum,
  author = {Shiyue Zhang and Asli Celikyilmaz and Jianfeng Gao and Mohit Bansal},
  title = {{EmailSum: Abstractive Email Thread Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Lx218llUlU},
}
Figure