Focus on the Action: Learning to Highlight and Summarize Jointly for Email To-Do Items Summarization

Kexun Zhang, Jiaao Chen, Diyi YangACL 2022 · OpenReview

Abstract

Extracting actionable to-do items from long email threads is a critical task for personal productivity. However, existing models often struggle with high annotation costs, noise propagation in multi-stage pipelines, and factual hallucinations. This paper presents the LHS (Learning to Highlight and Summarize) framework, an end-to-end joint learning approach. By integrating structured "Action Graphs" with a dual-encoder architecture and employing a novel name-perturbation technique, LHS achieves state-of-the-art performance in generating faithful and succinct email summaries.

Introduction

Automatic email to-do generation aims to distill commitments from unstructured text into a clean, actionable list. Despite its utility, the field faces three primary hurdles. First, human-labeled data for identifying "actionable" sentences is expensive to acquire. Second, traditional two-stage pipelines—which extract a sentence before summarizing it—often suffer from noise propagation, where errors in the first stage lead to failures in the second. Finally, models frequently "hallucinate" details, such as assigning tasks to the wrong person or inventing names not present in the source text.

Overview of the LHS workflow from email text to action graph and final to-do prediction.

To address these issues, the LHS framework moves toward a joint learning paradigm that captures the underlying logic of "who is doing what" through structured representations.

Methodology

Action Graph Construction

To capture the relational structure of an email, the framework utilizes OpenIE to extract subject-predicate-object triplets. These triplets are transformed into an Action Graph, where nodes represent entities and actions, and edges define their relationships. This structured approach ensures the model maintains a clear mapping of responsibilities.

Dual-Encoder Architecture

The system employs two distinct encoders: a standard transformer (BART) to process raw text and a Graph Attention Network (GAT) to encode the structured Action Graph. This allows the model to leverage both linguistic context and relational logic.

The LHS architecture featuring dual encoders for text and action graphs, and a multi-scale decoder.

Joint Learning and Soft Guidance

Instead of relying on manual labels, LHS uses an unsupervised "oracle" based on ROUGE scores to identify salient sentences and actions during training. The model learns to predict these "highlights" as an auxiliary task. During decoding, the system uses these highlight probabilities to "nudge" the decoder's attention toward relevant snippets, preserving context while filtering noise.

Name Perturbation

Analysis revealed that 35% of hallucinations involve named entities. To combat this, the authors introduced a name-perturbation technique, replacing person names with random common names during the final training stages. This forces the model to learn the functional role of an entity rather than memorizing specific names from the training set.

Experimental Evaluation

Datasets

The model was primarily evaluated on SmartToDo, a dataset of ~10,000 emails derived from the Avocado Email corpus. To test generalization, the researchers used EnronEmail, a smaller, out-of-domain set of 50 manually annotated emails.

Results

LHS demonstrated superior performance across both quantitative and qualitative metrics:

  • Quantitative: On the SmartToDo dataset, LHS achieved a ROUGE-L score of 0.692, significantly outperforming the standard BART baseline (0.669) and the previous state-of-the-art (0.630).
  • Zero-Shot Generalization: When tested on the Enron dataset without domain-specific training, LHS maintained a ROUGE-L of 0.614, indicating robust transferability.
  • Qualitative: Human evaluations confirmed that LHS produced summaries that were more factual, succinct, and informative than those generated by baseline models.

Conclusion

The LHS framework demonstrates that joint learning and structured action modeling are effective strategies for reducing hallucinations and improving the faithfulness of email to-do generation. By moving from hard filtering to soft guidance via highlight probabilities, the model maintains critical context while focusing on actionable content. Future work will explore extending this framework to multi-party conversations and more complex logical dependencies where objects are separated from their actions across multiple sentences.

Summary updated

BibTeX
@inproceedings{zhang2022focus,
  author = {Kexun Zhang and Jiaao Chen and Diyi Yang},
  title = {{Focus on the Action: Learning to Highlight and Summarize Jointly for Email To-Do Items Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=-f1KbPBB3I},
}
Figure