Evaluating the priority of email using machine learning

Xuan Long, Tisenko Victor Nikolaevich, Nguyen Lam, T.t. Thuong, Nguyen Anh Khoa Dam, Drew Conway, John White, Tingmin Wu, Sheng Wen, Yang Xiang, Wanlei Zhou, M Chandrasekaran, K Narayanan, Upadhyaya, F Toolan, J Carthy, Noor Jameel, M Ghazi, E Loay, George, S Nizamani, N Memon, M Glasdam, D Nguyen, T Kathsirvalavakumar, K Kavitha, R Palaniappan, Yong Fang, Cheng Zhang, Cheng Huang, Liang Liu, Yue Yang, ' Sa, Abdullah Al-Saaidah, B LeoInternational Journal of Emerging Trends in Engineering Research 2020 · DOI

TL;DR

This paper presents a machine learning framework designed to automatically sort the "wheat from the chaff" in your inbox by classifying Vietnamese emails as either important or unimportant. By combining social feature analysis (like sender frequency) with Natural Language Processing (NLP) and supervised learning, the authors achieved high classification accuracy, particularly with the Random Forest algorithm.

What Problem Does This Paper Address?

We are all familiar with "email fatigue." As the volume of digital communication grows, users struggle to identify which messages require immediate action and which are merely background noise. While "Priority Inbox" features exist in major platforms like Gmail, building effective models for specific languages (like Vietnamese) and individual user behaviors remains a challenge.

The authors aim to bridge this gap by creating a system that doesn't just look at keywords, but also considers the social context of an email—who sent it, how often you reply to them, and whether the email is part of an ongoing thread.

Methods

The researchers employed a multi-stage pipeline to evaluate email priority:

  1. Preprocessing & Tokenization: Since the focus is on Vietnamese text, they used specialized tools (ViTokenizer and VnTokenizer) for word segmentation.
  2. Feature Extraction: They extracted four primary fields: Subject, From, Date, and Body. They then calculated five specific "weight vectors":
  3. Sender Weight: Frequency of the sender's address in the training set.
  4. Response Weight: How often the user actually replies to that sender.
  5. Thread Weight: The activity level and duration of the email thread.
  6. Term/Message Weights: The importance of specific terms calculated via TF-IDF (Term Frequency-Inverse Document Frequency).
  7. Clustering: They used the K-means algorithm to cluster keywords into three distinct groups to better understand the distribution of "important" vs. "unimportant" terminology.
  8. Classification: Finally, they compared three supervised machine learning models to perform the actual binary classification: K-Nearest Neighbors (KNN), Random Forest, and Logistic Regression.

Datasets

The study used a real-world dataset consisting of 61,733 emails collected from 17 different users (retrieved via Google Takeout in Mbox format).

  • Domain: All emails were from the @fpt.edu.vn domain.
  • Language: The authors specifically filtered the dataset to focus on Vietnamese-language emails.
  • Labels: Out of the total, 20,054 emails were labeled as "important."

Key Insights and Ideas

  • Importance is Subjective: One of the most critical takeaways is that "importance" is not universal. The model's performance varied significantly across the 17 users. This suggests that a one-size-fits-all global model is less effective than a personalized model trained on an individual's specific habits.
  • Social Context Matters: The paper emphasizes that an email's priority isn't just about the text. Features like the "frequency of responses" between a sender and receiver are paramount in determining a strong social connection, which usually correlates with high importance.
  • The Power of Ensembles: Random Forest consistently outperformed simpler models like Logistic Regression. This is likely because Random Forest can better handle the non-linear relationships between disparate features like "time of day" and "specific keyword frequency."

Key Results

The Random Forest algorithm emerged as the clear winner across all metrics:

  • Average AUC (Area Under Curve): 0.854
  • Average F1-Score: 0.795
  • Average Recall: 0.797

In comparison, KNN achieved an AUC of 0.746, and Logistic Regression trailed at 0.668. Interestingly, for some specific users, the Random Forest model was exceptionally accurate, reaching an AUC of 0.953, while for others with less consistent habits, it dipped as low as 0.676.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Data Imbalance: The accuracy of the classification depends heavily on having a balanced set of important vs. unimportant emails for each user. Users with very few "important" emails provided less "signal" for the model to learn from.
  • Linguistic Nuance: While word segmentation helped, the habits of using specific words vary wildly between individuals.
  • Next Steps: The researchers plan to integrate Deep Learning algorithms and more complex behavioral profile analysis to further refine the system's ability to adapt to unique user habits over time.

Summary updated

Abstract

E-mail is an application that is widely used worldwide to help shorten the time, distance, and cost of the mailing process.In this paper, we propose a method of evaluating the role and importance of emails based on the process of analyzing and evaluating the content of emails.Our approach is based on Natural Language Processing and machine learning algorithms to classify emails into important or unimportant.Accordingly, our method is based on 2 main techniques: i) Processing and clustering of keywords in email.For this process, we will use data clustering algorithms.ii) Evaluating email content.For this process, we will use a supervised machine learning algorithm.The research results shown in the paper will provide email systems with a way to identify and classify important emails in order to assist users in sending and receiving emails.
BibTeX
@article{long2020evaluating,
  author = {Xuan Long and Tisenko Victor Nikolaevich and Nguyen Lam and T.T. Thuong and Nguyen Anh Khoa Dam and Drew Conway and John White and Tingmin Wu and Sheng Wen and Yang Xiang and Wanlei Zhou and M Chandrasekaran and K Narayanan and Upadhyaya and F Toolan and J Carthy and Noor Jameel and M Ghazi and E Loay and George and S Nizamani and N Memon and M Glasdam and D Nguyen and T Kathsirvalavakumar and K Kavitha and R Palaniappan and Yong Fang and Cheng Zhang and Cheng Huang and Liang Liu and Yue Yang and ' Sa and Abdullah Al-Saaidah and B Leo},
  title = {{Evaluating the priority of email using machine learning}},
  journal = {International Journal of Emerging Trends in Engineering Research},
  year = {2020},
  doi = {10.30534/ijeter/2020/233892020},
}
Figure