TL;DR
This paper introduces the Enron People Assignment (EPA) dataset, a novel resource for automatically identifying who is responsible for tasks mentioned in emails. They tackle the challenge of "addressee tagging" in noisy, real-world email communication, crucial for building intelligent email clients that can help users manage their workload.
What Problem Does This Paper Address?
In our daily deluge of emails, tasks are often buried within conversations. Imagine an email client that could automatically highlight tasks assigned to you and remind you to complete them. To build such a system, we need to solve two core problems: first, detecting the tasks themselves within the email text, and second, identifying which individuals mentioned or implied in the email are actually responsible for carrying out those tasks. This paper focuses on the latter, a problem they call "addressee tagging" or "addressee recognition." The challenge is amplified by the informal and often implicit nature of email communication, where responsibilities aren't always explicitly stated.
Methods
The core of this work is the creation and analysis of the EPA dataset. They leverage the Enron email corpus, pre-processing it to extract tasks as single sentences. The crucial step is then annotating these tasks with the responsible individuals. For each identified task, they consider the sender, "To," and "Cc" recipients as potential candidates. The problem is framed as a binary classification task for each (email, task, recipient) tuple: is this person responsible for this task? They developed an annotation process where crowd-workers were trained to identify responsible parties, even when not explicitly mentioned, relying on context and implicit references. They also experimented with a logistic regression model using handcrafted features, focusing on token replacement and out-of-sentence information.
Datasets
- Enron Email Dataset: Used as the source for extracting tasks and creating the EPA dataset.
- Avocado Research Email Collection: Used for comparison and as a training set for their experimental model. The authors note that their model trained on Avocado performed less well on the Enron dataset, highlighting the domain-specific challenges.
- Enron People Assignment (EPA) Dataset: A newly introduced benchmark dataset containing addressee assignment annotations for 15,649 (email, task, recipient) tuples.
Key Insights and Ideas
- Implicit Responsibility is Common: A significant finding is that identifying responsible parties often relies on implicit cues rather than explicit mentions, making this a challenging problem for automated systems.
- Email Noise is a Major Hurdle: The inherent messiness of email formatting and sentence segmentation significantly impacts the reliability of task extraction and, consequently, addressee tagging.
- Domain Matters: A model trained on one email corpus (Avocado) doesn't necessarily transfer well to another (Enron), suggesting that domain-specific adaptations are crucial for building robust email task management systems.
- "Nobody" is a Valid Assignment: The dataset acknowledges that sometimes, no one in the recipient list is responsible for a detected task, which is an important consideration for accurate assignment.
Key Results
- The EPA dataset achieved an inter-annotator agreement (Krippendorff's alpha) of 0.6123, which is considered moderate. This is lower than the 0.7854 achieved on the Avocado dataset, attributed to Enron's less clean formatting.
- A baseline model trained on the Avocado dataset achieved a precision of 0.6917 and recall of 0.8927 for single recipients on the Enron dataset, demonstrating reasonable performance but with room for improvement.
- The "every recipient" baseline (assuming everyone is responsible) performed poorly, especially in multi-recipient scenarios, highlighting the need for more sophisticated assignment logic.
Limitations and Future Work
- Annotation Noise: The authors acknowledge that the Enron dataset's formatting issues lead to more annotation noise compared to cleaner datasets like Avocado.
- Pre-processing Burden: Future work could involve more advanced pre-processing to reduce the burden on annotators and improve agreement.
- Extending to Other Deictic Tasks: The authors propose extending this work beyond addressee assignment to include temporal and location assignment for tasks, using temporal expression and location taggers.
- Incorporating Annotator Reliability: Future models could leverage annotator reliability metrics to filter data and improve training.
Summary updated