Introduction
Automating help-desk responses is a complex task because customer queries vary significantly in their linguistic requirements. While some emails can be answered with generic redirections, others require specific technical details. Traditional automation tools often rely on a single strategy—either document retrieval or statistical prediction—but no single method is universally effective. This paper, presented at AAAI 2007, introduces a meta-learning architecture that evaluates multiple response-generation strategies and selects the one most likely to succeed for a specific query.
Methodology
The proposed system employs a two-tier architecture. The first tier consists of four base strategies:
- Doc-Ret (Document Retrieval): Matches new requests to historical request-response pairs.
- Doc-Pred (Document Prediction): Maps request terms to clusters of previous responses.
- Sent-Pred (Sentence Prediction): Operates at the sentence level to assemble responses from multiple sources.
- Sent-Hybrid: A hybrid method that uses retrieval to supplement sentence prediction when uncertainty is high.
The second tier is a meta-learner that serves as a selection mechanism. Since confidence scores from different algorithms (e.g., cosine similarity vs. probability distributions) are not directly comparable, the meta-learner uses unsupervised clustering—specifically the Snob program—to calibrate these scores against historical performance.
By clustering past "experiences," the system learns to associate specific patterns of confidence scores with expected precision and recall. When a new email arrives, the meta-learner calculates confidence scores for all base methods, identifies the most similar historical cluster, and selects the strategy predicted to yield the best results.
Experimental Results
The approach was evaluated using a real-world corpus of 30,000 email dialogues from Hewlett-Packard (HP). The study focused on a sub-corpus of 6,659 two-turn dialogues, restricted to concise responses of 15 lines or fewer.
The meta-learning approach significantly outperformed random selection and remained competitive with theoretical "Gold" standards that have perfect knowledge of the best strategy. A key finding was the system's effectiveness as a high-precision filter. By setting a threshold where the system only provides an automated response if the estimated precision is $\ge 0.8$, the meta-learner achieved an actual average precision of 91-92%.
Under these conditions, the system successfully automated approximately 57% of all requests. This represents a substantial potential reduction in human workload with minimal risk of delivering incorrect information.
Conclusion
The core insight of this research is that algorithm confidence is relative and must be calibrated. By using unsupervised clustering to translate disparate confidence scores into a common language of expected performance, the meta-learner identifies the optimal granularity and strategy for each query. This approach allows help-desk systems to achieve high precision by knowing when to automate a response and when to defer to a human operator. Future work may explore extending this meta-learning framework to more complex, multi-step troubleshooting guides and testing its generalizability across different corporate domains.
Summary updated