Abstract
Extracting actionable to-do items from long email threads is a critical task for personal productivity. However, existing models often struggle with high annotation costs, noise propagation in multi-stage pipelines, and factual hallucinations. This paper presents the LHS (Learning to Highlight and Summarize) framework, an end-to-end joint learning approach. By integrating structured "Action Graphs" with a dual-encoder architecture and employing a novel name-perturbation technique, LHS achieves state-of-the-art performance in generating faithful and succinct email summaries.
Introduction
Automatic email to-do generation aims to distill commitments from unstructured text into a clean, actionable list. Despite its utility, the field faces three primary hurdles. First, human-labeled data for identifying "actionable" sentences is expensive to acquire. Second, traditional two-stage pipelines—which extract a sentence before summarizing it—often suffer from noise propagation, where errors in the first stage lead to failures in the second. Finally, models frequently "hallucinate" details, such as assigning tasks to the wrong person or inventing names not present in the source text.
To address these issues, the LHS framework moves toward a joint learning paradigm that captures the underlying logic of "who is doing what" through structured representations.
Methodology
Action Graph Construction
To capture the relational structure of an email, the framework utilizes OpenIE to extract subject-predicate-object triplets. These triplets are transformed into an Action Graph, where nodes represent entities and actions, and edges define their relationships. This structured approach ensures the model maintains a clear mapping of responsibilities.
Dual-Encoder Architecture
The system employs two distinct encoders: a standard transformer (BART) to process raw text and a Graph Attention Network (GAT) to encode the structured Action Graph. This allows the model to leverage both linguistic context and relational logic.
Joint Learning and Soft Guidance
Instead of relying on manual labels, LHS uses an unsupervised "oracle" based on ROUGE scores to identify salient sentences and actions during training. The model learns to predict these "highlights" as an auxiliary task. During decoding, the system uses these highlight probabilities to "nudge" the decoder's attention toward relevant snippets, preserving context while filtering noise.
Name Perturbation
Analysis revealed that 35% of hallucinations involve named entities. To combat this, the authors introduced a name-perturbation technique, replacing person names with random common names during the final training stages. This forces the model to learn the functional role of an entity rather than memorizing specific names from the training set.
Experimental Evaluation
Datasets
The model was primarily evaluated on SmartToDo, a dataset of ~10,000 emails derived from the Avocado Email corpus. To test generalization, the researchers used EnronEmail, a smaller, out-of-domain set of 50 manually annotated emails.
Results
LHS demonstrated superior performance across both quantitative and qualitative metrics:
- Quantitative: On the SmartToDo dataset, LHS achieved a ROUGE-L score of 0.692, significantly outperforming the standard BART baseline (0.669) and the previous state-of-the-art (0.630).
- Zero-Shot Generalization: When tested on the Enron dataset without domain-specific training, LHS maintained a ROUGE-L of 0.614, indicating robust transferability.
- Qualitative: Human evaluations confirmed that LHS produced summaries that were more factual, succinct, and informative than those generated by baseline models.
Conclusion
The LHS framework demonstrates that joint learning and structured action modeling are effective strategies for reducing hallucinations and improving the faithfulness of email to-do generation. By moving from hard filtering to soft guidance via highlight probabilities, the model maintains critical context while focusing on actionable content. Future work will explore extending this framework to multi-party conversations and more complex logical dependencies where objects are separated from their actions across multiple sentences.
Summary updated