Extracting Latent Beliefs and using Epistemic Reasoning to Tailor a Chatbot

Amit Sangroya, C. Anantaram, Pratik Saini, Mrinal RawatIJCAI 2018 · DOI

TL;DR

This paper introduces a framework for building more empathetic and effective chatbots that handle customer complaints. By extracting latent beliefs and employing epistemic reasoning, the chatbot can tailor its responses to better align with the customer's presumed mental state, leading to more meaningful conversations.

What Problem Does This Paper Address?

Traditional chatbots, even those employing advanced deep learning techniques, often struggle to maintain relevant and empathetic conversations, especially when dealing with customer complaints. A key issue is the mismatch between the chatbot's understanding of the situation and the customer's underlying beliefs and emotional state. This can lead to frustrating interactions where the chatbot's responses feel mechanical or irrelevant, hindering its ability to resolve the customer's issue effectively. The paper aims to bridge this gap by enabling chatbots to infer and act upon a customer's "latent beliefs" – their presumed cognitive states and expectations.

Methods

The proposed framework combines several key components:

  • Complaint Categorization: A Recurrent Neural Network (RNN), specifically a Long Short-Term Memory (LSTM) model, is used to classify customer complaints into predefined categories (e.g., engine problems, transmission issues). This initial step helps in understanding the general nature of the complaint.
  • Fact Extraction: Dependency parsers are employed to extract factual information from the complaint description in the form of triples. This is further refined using a hand-crafted fact-assertion rulebase, a domain ontology, and information extraction patterns.
  • Latent Belief Extraction: A hand-crafted belief knowledge base evaluates the extracted facts, slot values, and complaint category to assert probable beliefs held by the customer. For instance, if a car complaint describes the engine failing, beliefs like "car was working before" or "engine is not working now" might be asserted.
  • Epistemic Rule Evaluation: These asserted beliefs, along with the extracted facts, trigger epistemic rules. These rules help the chatbot infer what the customer knows or expects about the current state of affairs. For example, if the customer believes the engine is dead, the chatbot can infer that the customer knows the car won't start.
  • Dialog Tailoring: The insights from epistemic rules are used to tailor the dialog flow. This involves dynamically adjusting a Finite State Machine (FSM) that governs the conversation. States that are redundant or irrelevant based on the customer's inferred beliefs can be skipped, and new, more relevant questions can be prioritized.

Datasets

The paper utilizes a dataset of 61,252 car complaints from the website http://www.carcomplaints.com/. This dataset is used for training and evaluating the complaint categorization component.

Key Insights and Ideas

  • Beyond Surface-Level Understanding: The core idea is that effective complaint resolution requires understanding not just what the customer says, but what they believe and what they expect to happen.
  • Epistemic Reasoning for Empathy: By inferring a customer's "knowledge" and "beliefs," the chatbot can avoid asking obvious questions or making insensitive remarks, leading to a more empathetic interaction.
  • Dynamic Dialog Adaptation: The framework allows for a flexible dialog flow, where the chatbot can intelligently skip irrelevant parts of a predefined conversation script based on the customer's inferred state.
  • Domain Generality: While demonstrated with car complaints, the underlying architecture is presented as generic and applicable to other domains, such as hardware or software issue complaints.

Key Results

The paper reports a test accuracy of 68.47% for complaint categorization using a Vanilla LSTM model. The authors state that the beliefs and epistemic rules successfully helped tailor the dialog to customer expectations, making it more relevant.

Limitations and Future Work

  • Hand-crafted Knowledge Bases: The reliance on hand-crafted belief and epistemic rulebases, as well as domain ontologies, can be labor-intensive and may limit scalability to new domains without significant effort.
  • Accuracy of Belief Extraction: The accuracy and robustness of latent belief extraction are crucial for the framework's success, and further research might be needed to improve this aspect.
  • Future Work: The authors suggest that the approach is generic and can be applied to other domains. Further exploration into automating the creation of knowledge bases and improving the accuracy of belief inference would be valuable future directions.

Summary updated

BibTeX
@inproceedings{sangroya2018extracting,
  author = {Amit Sangroya and C. Anantaram and Pratik Saini and Mrinal Rawat},
  title = {{Extracting Latent Beliefs and using Epistemic Reasoning to Tailor a Chatbot}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/860},
}
Figure