TL;DR
This paper tackles the challenge of automatically summarizing email threads, which are inherently dialogic and collaborative. They propose an extractive summarization approach that leverages email-specific features, demonstrating that these features significantly improve summarization performance compared to generic text summarization techniques.
What Problem Does This Paper Address?
Summarizing email threads presents a unique challenge compared to summarizing single documents like news articles. Email threads are asynchronous, multi-party conversations where participants build upon previous messages, often through direct quotes. This dialogic structure, combined with the lack of physical co-presence, means that understanding and summarizing an email thread requires capturing the interactional aspect and identifying relevant context from prior messages. Existing summarization techniques often treat text as a monolithic block, failing to account for the nuanced structure and collaborative nature of email exchanges.
Methods
The authors employ an extractive summarization approach, meaning they identify and select important sentences from the original thread to form the summary. Their core contribution lies in developing and evaluating a set of features that go beyond generic text summarization.
They start with a "basic" feature set that includes standard features applicable to any text genre, such as sentence position within the thread and similarity to the thread's centroid (average TF-IDF vector of all sentences).
Next, they introduce "basic+" features that account for the thread's message structure. This includes the ordinality of a message within the thread and the relative position of a sentence within its message.
Finally, they propose "full" features that are specific to email communication. These include:
- Subject similarity: Overlap between the subject of the first message and the sentence's content.
- Response/Recipient counts: Number of direct responses to a message and the number of recipients.
- Quotation detection: Whether a sentence follows a quoted portion of a previous message.
These features are used to train a machine learning classifier (Ripper) to identify sentences that should be included in the summary.
Datasets
The authors created a corpus of 96 email threads from the board of the student organization of the ACM at Columbia University. These threads primarily concerned event planning. Each thread was manually summarized by two annotators. The summaries were then used to label sentences within the threads as either "summary-worthy" (marked "Y") or not (marked "N"). Approximately 26% of sentences were marked "Y".
Key Insights and Ideas
- Email is a distinct genre: The inherent dialogic and collaborative nature of email threads requires specialized summarization techniques.
- Email-specific features matter: Features that capture the structure of email communication (like quoted text and message flow) significantly improve summarization performance over generic text features.
- Human summarization is variable: The study highlights considerable variation between human annotators, suggesting that machine learning models might perform better by learning from a single, consistent annotator rather than trying to reconcile diverse summarization styles.
- Quoted text is a valuable signal: Features related to quoted material are identified as important for summarization.
Key Results
The experiments showed that using the "full" feature set (including email-specific features) significantly outperformed the "basic" feature set and a simple centroid-based baseline. For annotator DB, the F-measure increased from 0.272 (basic) to 0.497 (full). For annotator GR, the improvement was even more pronounced, with the full feature set outperforming the centroid baseline and achieving an F-measure of 0.352. Combining the results of two annotators did not yield better results, suggesting that the inconsistency in human summarization styles hindered the learning process.
Limitations and Future Work
The authors acknowledge that their approach is primarily extractive and that the presentation of the summary needs to account for the dialogic structure. They also note that the results for annotator GR were significantly worse than for DB, speculating that GR was less consistent in her summarization choices, making it harder for the machine learner to find patterns.
Future work includes:
- Qualitative error analysis: To understand why certain summaries are better than others and to improve human annotator guidelines.
- Learning predictors for other thread aspects: Such as thread category and question-answer pairs, to generate more "complete" summaries.
- Human feedback evaluation: To assess the quality of the generated summaries from a user perspective.
- Exploring other summarization techniques: Beyond sentence extraction.
Summary updated