Modern email traffic is overwhelmingly machine-generated, consisting of receipts, newsletters, social media alerts, and spam. For major providers like Yahoo Mail, accurately identifying the small fraction of "human" (personal) emails is critical for features like priority notifications and "People View." This research, presented at AAAI-22, details a transition from traditional feature engineering to a deep learning approach that treats user behavior as a training signal rather than an input feature, effectively solving the "sender cold-start" problem.
The Challenge: Sender Cold-Start and Sequence Modeling
Previous production systems relied on logistic regression models using aggregated features at the SMTP (sender) level. This approach suffered from two major flaws:
- Sender Cold-Start: New senders lacked historical data, leading to poor classification for approximately 70% of traffic.
- Lack of Sequence Modeling: "Bag of words" approaches ignored the structural and temporal nuances of language captured by modern deep learning.
Proposed Methodology: The Multi-Component CNN
The researchers developed a "Full Model" architecture composed of four specialized CNN-based sub-models designed to process different aspects of an email:
- Content Model: Processes the subject line and email body.
- Sender Model: Analyzes the sender’s email address (using letter-trigrams) and display name.
- Action Model: Predicts the probability of a user opening or deleting a message based on content, baking behavioral knowledge into the model weights.
- Salutation Model: Detects explicit salutations (e.g., "Dear John"), a strong indicator of human-to-human communication.
Engineering for Scale
To meet production latency requirements (processing 30 million messages per hour), the team used a custom vocabulary and 64-dimensional embeddings. This resulted in a model four times smaller than those using heavy pre-trained embeddings like GloVe, with no loss in accuracy. A "rectification" strategy was employed to combine sub-models, allowing the Sender and Action models to influence the final score only when their confidence exceeded 0.99.
Key Insights: Behavior as a Label
A pivotal innovation of this work is treating user behavior as a label rather than a feature. By training a model to predict user behavior (like opening or deleting) based on message content, the system can infer the nature of an email even from a brand-new sender.
Furthermore, the authors found that their CNN-based model outperformed fine-tuned BERT models. While BERT is often limited to 512 tokens, emails frequently contain machine-generated signatures (like "Unsubscribe" links) at the very end of long messages. The CNN was configured with a 2000-token window, allowing it to capture these vital signals that BERT missed.
Results and Production Impact
The full model significantly outperformed the legacy logistic regression baseline and BERT-based alternatives:
- Accuracy: Adjusted recall increased from 70.5% to 78.8%, while precision improved from 94.7% to 96.0%.
- Efficiency: The system is currently deployed in Yahoo Mail 6, performing inference on 30 million messages per hour.
Despite these gains, challenges remain regarding class imbalance—since human email represents only 3-5% of traffic—and the privacy constraints that limit the availability of manually labeled "gold standard" datasets. Future work may involve refining the architecture with additional sub-models focusing on HTML structure or attachments.
Summary updated