TL;DR
This paper presents a production-ready, end-to-end AI system for automatically routing helpdesk email tickets to the correct technical teams. By combining an ensemble of machine learning classifiers with a configurable rule engine, the system achieves human-level accuracy (~90%) at scale, saving over 50,000 man-hours annually for major IT service providers.
Problem Statement
In large-scale IT service delivery, "ticket dispatch"—the process of reading an incoming email and assigning it to the right "resolver group" (e.g., Database Admin, Network Security, Hardware Repair)—is a massive bottleneck.
While it sounds like a standard text classification problem, three real-world challenges make it "push the limits":
- Scale and Complexity: Large companies can have over 500 resolver groups with overlapping responsibilities.
- Organizational Volatility: Helpdesk teams frequently merge, split, or rename. A model trained yesterday might point to a group that doesn't exist today.
- The "Long Tail": A few groups receive the vast majority of tickets, while hundreds of others receive only a handful. Standard machine learning models struggle to learn from these sparse "long tail" examples.
Proposed Architecture
The authors propose a hybrid architecture that balances the power of deep learning with the precision of expert-defined rules.
- Ensemble Classification Engine: The system evaluates traditional models (like Linear SVM) and deep learning models (MLP, CNN, and LSTM). In production, an ensemble (typically SVM + MLP) is used to maximize both accuracy and coverage.
- Configurable Rule Engine: This acts as a "safety net" to handle domain-specific content and ensure business continuity. For example, if a team is renamed, a simple rule can redirect the ML output without needing a full model retrain.
- Long Tail Strategy: The ML model focuses on the top ~20% of groups that handle ~98% of the volume. The remaining "long tail" is handled via the rule engine.
- Automated Retraining: To combat "concept drift," the system uses a sliding window retraining strategy, training on the last 90 days of data every 7 days.
Datasets
The system was evaluated on three real-world datasets from major service providers (Telecom and Supply Chain/Logistics):
- Dataset A: 11,562 tickets, 70 resolver groups.
- Dataset B: 423,343 tickets, 403 resolver groups.
- Dataset C: 712,320 tickets, 428 resolver groups.
These datasets consist of raw, informal email text (subject and body) rather than clean, structured web forms.
Key Results and Insights
- Human-Level Performance: By mining audit logs, researchers found human dispatchers are roughly 85% accurate. This AI system consistently hits or exceeds this mark, achieving ~90% accuracy with over 90% coverage across production datasets.
- Simplicity and Efficiency: Simple models like Linear SVM and Multi-Layer Perceptrons (MLP) often performed as well as complex CNNs while being faster to deploy. On Dataset C, optimization techniques reduced model size by 99% and training time by 93.5%.
- The Necessity of Rules: In production, a rule engine is vital for handling sudden business changes (like team mergers) that ML cannot adapt to instantly.
- Business Impact: The system processes over 90,000 emails per month, translating to a net saving of more than 50,000 man-hours per year across three deployed sites. The production system handles ~286 requests per second, far exceeding human throughput.
Limitations and Future Work
Currently, the system only processes email text and does not analyze screenshots, logs, or PDF attachments. Future work includes implementing a more sophisticated "active learning" strategy to identify high-value misclassified tickets for retraining and improving the system's response to sudden shifts in terminology or new types of IT outages.
Summary updated