TL;DR
This paper introduces Active Feature Probing (AFP), a novel method for dynamically generating clarification questions to helpdesk users to efficiently reach the correct recommendation. Unlike static approaches, AFP actively uses user input to refine its questions, and its probabilistic model with data augmentation makes it robust to incomplete or error-prone helpdesk databases.
What Problem Does This Paper Address?
Customer service is crucial, and helpdesk databases store past interactions to improve service quality. A common use case is finding existing recommendations for a new customer problem. However, customers often provide incomplete or inaccurate information, making direct database matching difficult. Manually creating clarification questions is impractical for large, frequently updated databases. This paper tackles the challenge of automatically generating a minimal set of questions to reach an appropriate recommendation, ensuring all user-provided information is utilized and the system is resilient to database imperfections.
Methods
The core of the proposed solution is Active Feature Probing (AFP). AFP operates dynamically:
- Probabilistic Question Selection: AFP uses a probabilistic model to estimate the "informativeness" of potential questions. It calculates the entropy of the class label (recommendation) given the currently known features. The goal is to select questions that maximally reduce this uncertainty. This is achieved by maximizing the mutual information between a feature (question) and the class label, given the observed features.
- Data Augmentation: To overcome the challenge of estimating probabilities with limited or incomplete data, AFP employs a data augmentation technique. This method generates synthetic "augmented samples" that mimic the known features of the new case and synthetically fills in the unknown features based on distributions learned from the helpdesk database. This helps to avoid overfitting and makes the model robust to database errors.
- Dynamic Interaction: AFP presents the user with a ranked list of questions and potential recommendations. As the user provides answers or additional information, this input is immediately incorporated into the set of known features, triggering a new round of question and recommendation ranking. This iterative process continues until a recommendation's probability exceeds a certain threshold or the user selects one.
Datasets
- Synthetic Dataset: A dataset of 1,000,000 documents from news reports, treated as cases with 35 categories (recommendations) and 500 informative words as features. This dataset was used to study the impact of various parameters like the number of categories, features, and probes.
- Real Helpdesk Database: A dataset from a research university's technical support group, containing approximately 340,865 cases with 25 recommendations and 1000 extracted features. This dataset was used for user studies and simulations.
Key Insights and Ideas
- Dynamic Questioning is Key: The most significant insight is that actively using user feedback to dynamically adjust the questioning strategy is far more effective than static approaches like traditional decision trees. This allows the system to adapt to the specific information provided by each user.
- Probabilistic Robustness: Relying on a probabilistic model, especially when combined with data augmentation, makes the system resilient to the inherent incompleteness and potential errors in real-world helpdesk databases. This is a critical advantage over methods that might overfit to a few similar, potentially flawed, cases.
- Efficiency Through Informativeness: By focusing on questions that maximally reduce uncertainty (i.e., have high mutual information with the recommendation), AFP aims to reach a conclusion with a minimal number of probes, saving both user and system time.
Key Results
- Superior Accuracy: In user studies, AFP achieved significantly higher accuracy (e.g., 82.5% vs. 57.0% for DT) and required fewer questions on average (e.g., 6 vs. 11.1 for DT) compared to Decision Trees (DT), Empirical Dynamic Decision Trees (EMP), and Case-Based Reasoning (CBR).
- Robustness to Complexity: AFP maintained high accuracy across test cases of varying complexity (low, medium, high), outperforming other methods in all categories.
- Impact of User Input: Experiments demonstrated that initial information, corrections, and additional information provided by users significantly boost AFP's accuracy, highlighting the importance of its interactive nature.
- Scalability: Execution time for AFP was found to increase linearly with the number of features, recommendations, and cases, with individual case processing being very fast (<0.1 seconds), making it suitable for interactive use.
Limitations and Future Work
- Feature Extraction: The current approach assumes features are pre-extracted. Future work includes investigating Natural Language Processing (NLP) techniques for automatic feature extraction.
- Algorithm Efficiency: While the current algorithm is efficient, generating augmented samples can be computationally intensive for extremely large databases. Future research aims to explore more efficient sampling or approximation methods.
- Sequential Dependencies: The paper acknowledges that questions often have sequential dependencies. The authors propose an extension to handle this by incorporating partial orders provided by domain experts, using a breadth-first search approach to prioritize questions.
Summary updated