EmailSum: Abstractive Email Thread Summarization

Shiyue Zhang, Asli Celikyilmaz, Jianfeng Gao, Mohit Bansal

Abstract

Recent years have brought about an interest in the challenging task of summarizing conversation threads (meetings, online discussions, etc.). Such summaries help analysis of the long text to quickly catch up with the decisions made and thus improve our work or communication efficiency. To spur research in thread summarization, we have developed an abstractive Email Thread Summarization (EmailSum) dataset, which contains human-annotated short (<30 words) and long (<100 words) summaries of 2549 email threads (each containing 3 to 10 emails) over a wide variety of topics. We perform a comprehensive empirical study to explore different summarization techniques (including extractive and abstractive methods, single-document and hierarchical models, as well as transfer and semisupervised learning) and conduct human evaluations on both short and long summary generation tasks. Our results reveal the key challenges of current abstractive summarization models in this task, such as understanding the sender's intent and identifying the roles of sender and receiver. Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task. Hence, we emphasize the importance of human evaluation and the development of better metrics by the community.

Introduction

While automated summarization has seen massive progress in domains like news articles and chat logs, email threads remain a significant challenge. Emails possess a unique, rigid structure—including headers, greetings, and signatures—and often involve technical jargon and multiple participants interacting over extended periods.

The primary bottleneck in this field has been the lack of large-scale, high-quality data. Previous email datasets were often limited to fewer than 50 threads, making it difficult to train and evaluate modern deep-learning models. Without a robust benchmark, it has been unclear whether models optimized for news (e.g., CNN/DailyMail) or dialogue (e.g., SAMSum) can effectively translate to the intent-heavy environment of professional email communication.

The EMAILSUM Dataset

To address this data scarcity, the authors introduced EMAILSUM, a dataset derived from the Avocado Research Email Collection and the W3C email corpus.

  • Scale: 2,549 email threads, each containing between 3 and 10 emails.
  • Annotations: Each thread features two human-written summaries: a short summary (under 30 words) focusing on the primary point, and a long summary (under 100 words) providing a narrative of the entire discussion.
  • Semi-Supervised Support: The dataset includes 8,594 additional unlabeled threads to facilitate research into semi-supervised learning techniques.

Methodology

The researchers evaluated several summarization paradigms to establish a baseline for the task:

  1. Extractive Baselines: These include simple heuristics like "Lead-1-Email" (taking the subject line and the first sentence of every email) and graph-based methods like TextRank.
  2. Abstractive Baselines: The primary model used was T5 (Text-to-Text Transfer Transformer). The study compared a standard "T5-base" model, which treats the thread as a single document, against a specialized Hierarchical T5.
  3. Transfer & Semi-Supervised Learning: The authors explored pre-finetuning on news or dialogue datasets and utilized a self-training approach on the unlabeled data to boost performance.
The Hierarchical T5 architecture utilizes a token-level encoder and an email-level encoder to capture both local and global context.

As illustrated in Figure 1, the Hierarchical T5 model uses a dual-encoder structure. The token-level encoder processes individual words, while the email-level encoder uses mean pooling to aggregate these into representations that capture the flow of the entire thread.

Key Insights and Results

The "Intent" and "Role" Gaps

The study found that the primary failure of current models is not fluency, but a lack of deep understanding regarding sender intent. Models often struggle to distinguish between a request for a meeting and a broadcasted decision. Furthermore, role confusion is common; models frequently swap the identities of participants, which is critical in professional contexts where the seniority or role of the sender matters.

  • T5 Dominance: Simply finetuning T5-base as a single-document summarizer outperformed most extractive and complex hybrid models.
  • Hierarchical Benefits: The Hierarchical T5 model showed a slight advantage in generating long summaries, suggesting that explicit structural modeling is beneficial for longer outputs.
  • Domain Specificity: Transferring knowledge from news or chat datasets provided minimal improvements, highlighting that email summarization is a distinct domain requiring specialized data.

The Metric Problem

One of the most striking findings of the research is the weak correlation between standard automatic metrics and human judgment.

Pearson correlation coefficients demonstrate that ROUGE and BERTScore align poorly with human ratings for salience and faithfulness.

As shown in Figure 3, metrics like ROUGE and BERTScore show very low Pearson correlation coefficients with human ratings for salience and faithfulness. In some cases, models that achieved higher ROUGE scores by being more extractive were actually less preferred by human judges, who valued the abstractive, intent-focused nature of human-written summaries.

Conclusion

EMAILSUM provides a necessary foundation for advancing email thread summarization. However, the task remains difficult for current models, particularly in capturing participant roles and underlying intents. The research emphasizes that as long as the community relies on metrics like ROUGE, we may be optimizing for "copy-pasting" rather than true comprehension. Future work must focus on developing metrics that better align with human values and exploring more sophisticated ways to utilize unlabeled data without losing abstractive quality.

Summary updated

Abstract

Recent years have brought about an interest in the challenging task of summarizing conversation threads (meetings, online discussions, etc.). Such summaries help analysis of the long text to quickly catch up with the decisions made and thus improve our work or communication efficiency. To spur research in thread summarization, we have developed an abstractive Email Thread Summarization (EmailSum) dataset, which contains human-annotated short (<30 words) and long (<100 words) summaries of 2549 email threads (each containing 3 to 10 emails) over a wide variety of topics. We perform a comprehensive empirical study to explore different summarization techniques (including extractive and abstractive methods, single-document and hierarchical models, as well as transfer and semisupervised learning) and conduct human evaluations on both short and long summary generation tasks. Our results reveal the key challenges of current abstractive summarization models in this task, such as understanding the sender's intent and identifying the roles of sender and receiver. Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task. Hence, we emphasize the importance of human evaluation and the development of better metrics by the community. Our code and summary data have been made available at: https://github.com/ZhangShiyue/EmailSum
BibTeX
@inproceedings{zhang2021emailsum,
  author = {Shiyue Zhang and Asli Celikyilmaz and Jianfeng Gao and Mohit Bansal},
  title = {{EmailSum: Abstractive Email Thread Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Lx218llUlU},
}
Figure