Email answering assistance by semi-supervised text classification

Tobias SchefferIntell. Data Anal 2004

TL;DR

This paper tackles the challenge of automatically suggesting appropriate answers to common customer service emails. By framing this as a semi-supervised text classification problem, the authors demonstrate that both transductive SVMs and co-training can significantly improve answer prediction accuracy when labeled data is scarce, outperforming traditional supervised methods.

What Problem Does This Paper Address?

Companies receive a massive volume of emails, many of which contain repetitive questions. While fully automating email responses is often impractical, providing assistance to human responders by suggesting likely answers can significantly boost productivity. The core problem is to predict which of a predefined set of standard answers a user will choose to reply to an incoming email. This is complicated by the fact that emails can be phrased in many ways, and the frequency of certain questions can change over time (non-stationarity).

Methods

The authors frame the email answering problem as a semi-supervised text classification task. For each standard answer, they train a binary classifier to distinguish emails that should receive that answer from those that shouldn't.

Key techniques include:

  • Naive Bayes and Support Vector Machines (SVMs): These are used as baseline supervised classifiers. SVMs, particularly with TF-IDF features, show strong performance.
  • Transductive Support Vector Machines (TSVMs): This approach extends SVMs to leverage unlabeled data by optimizing the decision boundary for both labeled and unlabeled examples.
  • Co-training: This algorithm uses two classifiers trained on different, independent subsets of features. They iteratively label unlabeled data for each other, bootstrapping their performance.
  • Handling Non-Stationarity: The paper addresses the changing probability of answers over time by using a discounted moving average for prior probabilities.

Datasets

The primary dataset used is from the TELES European Internet Academy, an educational provider. This dataset consists of:

  • 528 inbound emails manually labeled with one of nine standard answers.
  • Corresponding outbound emails to infer reply relationships.

This dataset is used for both training and evaluating the proposed methods. No new benchmark dataset is introduced.

Key Insights and Ideas

  • Semi-supervised learning shines with limited data: The most striking insight is that both TSVM and co-training provide substantial performance gains only when the amount of labeled training data is very small. Beyond a certain threshold, their benefit diminishes or even leads to performance degradation.
  • SVMs are powerful for this task: Even without semi-supervised learning, SVMs demonstrate superior performance compared to Naive Bayes for identifying specific email questions, even with very few labeled examples.
  • Co-training's limitations are tied to assumptions: The paper provides evidence that co-training's effectiveness is hampered when its core assumptions (compatibility and independence of feature views) are violated. This violation is shown to increase with the size of the labeled dataset and the number of co-training iterations.
  • Practical integration is feasible: The research demonstrates that these machine learning techniques can be integrated into a real-world email assistance system (Responsio) without requiring explicit user feedback for learning.

Key Results

  • High AUC performance: SVMs achieved AUC performance between 80% and 95% even with as few as seven labeled positive examples for some categories.
  • TSVM and Co-training benefits:
    • For "general product inquiries," TSVM outperformed SVM when labeled samples were small (e.g., 16 positive, 22 negative examples).
  • For "server breakdown," co-training showed significant improvement with as few as 5 positive examples.
  • Performance degradation: When a large number of labeled examples were available (e.g., 50 positive examples), co-training performance deteriorated.
  • Computational cost: Both TSVM and co-training significantly increase computation time, from seconds to up to 20 minutes.

Limitations and Future Work

  • Inability to identify "no standard answer": The current approach cannot determine when an email doesn't fit any of the standard answers, nor can it estimate the probability of this scenario. This is due to the reliance on matching outbound emails to inbound ones and the assumption that outbound emails either match a standard answer or are semantically equivalent.
  • Difficulty with semantically equivalent answers: The system struggles to identify when a user's response is semantically equivalent to a standard answer but phrased differently.
  • Computational efficiency: The substantial increase in computation time for TSVM and co-training is a significant drawback for desktop applications, suggesting a need for efficiency improvements.
  • Future work could explore:
    • More robust methods for handling the "no standard answer" case.
  • Techniques to better capture semantic equivalence in responses.
  • Optimizing the efficiency of semi-supervised methods for real-time applications.

Summary updated

BibTeX
@article{scheffer2004email,
  author = {Tobias Scheffer},
  title = {{Email answering assistance by semi-supervised text classification}},
  journal = {Intell. Data Anal.},
  year = {2004},
}
Figure