Evaluating Chatbots to Promote Users' Trust - Practices and Open Problems

Biplav Srivastava, Kausik Lakkaraju, Tarmo Koppel, Vignesh Narayanan, Ashish Kundu, Sachindra JoshiarXiv 2023 · OpenReview

Abstract

Chatbots, the common moniker for collaborative assistants, are Artificial Intelligence (AI) software that enables people to naturally interact with them to get tasks done. Although chatbots have been studied since the dawn of AI, they have particularly caught the imagination of the public and businesses since the launch of easy-to-use and general-purpose Large Language Model-based chatbots like ChatGPT. As businesses look towards chatbots as a potential technology to engage users, who may be end customers, suppliers, or even their own employees, proper testing of chatbots is important to address and mitigate issues of trust related to service or product performance, user satisfaction and long-term unintended consequences for society. This paper reviews current practices for chatbot testing, identifies gaps as open problems in pursuit of user trust, and outlines a path forward.

The Blackbox Problem

The meteoric rise of Large Language Model (LLM) based chatbots has introduced a significant challenge: unpredictability. Unlike traditional rule-based systems, LLMs often behave like a "blackbox," where the same factual query can yield vastly different results based on subtle, seemingly irrelevant changes in the prompt.

Figure 1: Example of inconsistent chatbot behavior based on user identity.

A startling case study highlighted in this research shows that when asking for US state capitals, a chatbot's tone and content changed based solely on the user's name. If a name was associated with a specific racial stereotype, the bot became condescending or veered into unsolicited topics like civil rights. This lack of stability directly undermines user trust, proving that "correctness" alone is no longer a sufficient metric for success.

Beyond Traditional Testing

Current testing practices often fail to capture the nuances of human-AI collaboration. The paper surveys existing methods across three primary lenses:

  1. Software Engineering: Focuses on traditional unit, system, and acceptance testing to ensure the code runs as intended.
  2. AI Systems: Evaluates data privacy, provenance, and model performance, including robustness against adversarial attacks.
  3. Human Collaboration: Utilizes explainability (XAI) and Randomized Controlled Trials (RCTs) to measure the actual impact on the user.

Despite these efforts, there is no rigorous, industry-standard framework to guarantee that a chatbot will remain stable, unbiased, and trustworthy over long-term use.

A New Perspective: Chatbots as Dynamical Systems

To address the stability issue, the authors propose a novel theoretical framework: modeling chatbots as Dynamical Systems. By viewing a conversation as a sequence of states in a high-dimensional Markov Decision Process (MDP), researchers can apply control-theory concepts like stability and convergence.

In this model, a reliable chatbot should consistently "converge" to the correct answer regardless of minor variations in the prompt (exogenous input). If the bot's responses stay within a predefined set of "correct" or "safe" states, it is considered "forward-invariant." This mathematical approach allows for a more formal definition of reliability than simple accuracy scores.

The Stakeholder Gap

A major hurdle in chatbot evaluation is the misalignment of goals between different parties involved in the AI lifecycle. Developers, businesses, customers, and regulators all prioritize different metrics, making it difficult to create a single "trustworthy" standard.

Table 1: Comparison of stakeholder interests and evaluation metrics.

While developers focus on technical bugs and maintenance costs, customers are primarily concerned with ease of use and the reliability of the information provided. Bridging this gap requires a multi-faceted evaluation strategy that satisfies these diverse requirements simultaneously.

Open Problems and the Path Forward

The paper identifies five critical gaps in current chatbot development:

  • Lack of testing rigor: Current methods are often ad-hoc rather than systematic.
  • Reproducibility crisis: Because dynamic systems change over time, repeating tests is difficult.
  • Societal impact: Failure to understand long-term consequences of AI interaction.
  • Communication: Poor methods for informing end-users of a bot's limitations.
  • Inclusivity: A lack of testing for bias across diverse user demographics.

Moving forward, the authors advocate for the development of "Trust Certificates"—nutrition-label style certifications that communicate a bot's strengths and weaknesses—and the adoption of automated tools to perform "Monte Carlo" analysis on chatbot trajectories to identify potential points of failure before they reach the user.

Summary updated

Abstract

Chatbots, the common moniker for collaborative assistants, are Artificial Intelligence (AI) software that enables people to naturally interact with them to get tasks done. Although chatbots have been studied since the dawn of AI, they have particularly caught the imagination of the public and businesses since the launch of easy-to-use and general-purpose Large Language Model-based chatbots like ChatGPT. As businesses look towards chatbots as a potential technology to engage users, who may be end customers, suppliers, or even their own employees, proper testing of chatbots is important to address and mitigate issues of trust related to service or product performance, user satisfaction and long-term unintended consequences for society. This paper reviews current practices for chatbot testing, identifies gaps as open problems in pursuit of user trust, and outlines a path forward.
BibTeX
@misc{srivastava2023evaluating,
  author = {Biplav Srivastava and Kausik Lakkaraju and Tarmo Koppel and Vignesh Narayanan and Ashish Kundu and Sachindra Joshi},
  title = {{Evaluating Chatbots to Promote Users' Trust - Practices and Open Problems}},
  year = {2023},
  eprint = {2309.05680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.05680},
}
Figure