Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach

Todd Kulesza, Margaret M. Burnett, Simone Stumpf, Weng-Keen Wong, Shubhomoy Das, Alex Groce, Amber Shinsel, Forrest Bice, Kevin McIntoshIS-EUD 2011 · OpenReview

TL;DR

Intelligent assistants are becoming more critical, but users lack systematic ways to identify their mistakes. This paper introduces WYSIWYT/ML, a human-AI partnership that empowers end-users to systematically test these assistants, significantly improving bug detection efficiency and effectiveness compared to traditional ad-hoc methods.

What Problem Does This Paper Address?

As intelligent assistants (like email sorters or safety monitors) take on more critical tasks, ensuring their reliability becomes paramount. However, current methods for users to assess these assistants are largely "ad hoc" – relying on serendipitous observation of errors. This is problematic because machine learning models are inherently imperfect, and mistakes in critical applications can have significant consequences. Users need a structured way to understand where and when their assistants might be making errors, especially as these assistants learn and adapt over time. The paper addresses the gap in systematic, user-driven assessment of these evolving AI systems.

Methods

The core of the paper is the WYSIWYT/ML (What You See Is What You Test for Machine Learning) approach. It's a human-computer partnership designed to help end-users systematically test intelligent assistants. The system offers four key functions:

  1. Prioritization: WYSIWYT/ML advises users on which predictions are most likely to be wrong, helping them focus their testing efforts. This is achieved through three candidate methods:
  2. Confidence: Prioritizes based on the assistant's certainty about its prediction.
  3. Similarity: Identifies "oddball" messages that are dissimilar to the training data.
  4. Relevance: Focuses on messages with fewer "useful" words for prediction.
  5. Contribution of Tests: When a user tests a message (marks it as correct or incorrect), WYSIWYT/ML automatically infers similar judgments for other messages. This is the "partnership" aspect, where the computer contributes tests based on user input.
  6. Coverage Measurement: The system tracks how much of the assistant's reasoning has been "tested" by the user and inferred tests. This provides a measure of how thoroughly the assistant's behavior has been assessed.
  7. Monitoring for Changes: WYSIWYT/ML monitors for shifts in the assistant's behavior over time, alerting users when previously tested areas might need re-evaluation due to the assistant's learning or new data.

The approach is inspired by the WYSIWYT methodology for spreadsheets but adapts it for machine learning models, focusing on statistical properties rather than source code structure.

Datasets

  • 20 Newsgroups dataset: Used for training and evaluation. This dataset is a standard benchmark for text classification tasks.
  • Customized data sets: The authors created specific training and test sets from the 20 Newsgroups data, including a tutorial set, to familiarize participants with the system and evaluate its performance.

Key Insights and Ideas

  • Systematic testing is crucial for AI reliability: The paper strongly argues that ad-hoc testing is insufficient for critical AI applications, and a structured approach is necessary for users to build trust.
  • Human-AI partnership is key: WYSIWYT/ML demonstrates that by combining user judgment with automated inference, testing can be significantly amplified. The computer acts as a smart assistant to the user's testing efforts.
  • Prioritization methods matter: Different prioritization strategies (Confidence, Similarity, Relevance) highlight different types of bugs, suggesting that a combination might be most effective.
  • Coverage provides a sense of assurance: A measurable coverage metric helps users understand the extent of their testing and decide when they have enough confidence in the assistant.
  • "Maybe" marks need careful interpretation: Pilot studies revealed that users might repurpose "maybe" marks for reminders rather than genuine uncertainty, impacting how the system infers tests.

Key Results

  • Efficacy (RQ1): Participants using WYSIWYT/ML found significantly more bugs than those using ad-hoc testing. The best variant (Confidence) led to participants finding nearly twice as many bugs.
  • Satisfaction (RQ2): Users rated WYSIWYT/ML treatments as more helpful than the control (ad-hoc) treatment, indicating a positive attitude towards systematic testing.
  • Efficiency (RQ3): WYSIWYT/ML significantly improved efficiency. Participants were able to test an average of 117 messages in just 10 minutes. The system scaled to cover 623 out of 1,448 messages with only 10 minutes of user effort, representing a 10-fold leveraging of user input.
  • Bug 20635: This specific "hard bug" was found by a much higher proportion of participants using the Relevance prioritization method, highlighting the complementarity of different strategies.

Limitations and Future Work

  • "Maybe" mark interpretation: The study noted that users' use of "maybe" marks for reminders rather than true uncertainty led to a high false rate for inferred disapprovals (X marks). Future work could explore better ways to handle these nuanced judgments, perhaps through secondary notation.
  • Coverage threshold for disapprovals: The current similarity threshold for inferring disapprovals might need adjustment to reduce false positives.
  • Empirical investigation of user willingness: While the paper targets users willing to invest effort, it acknowledges that a direct empirical test of whether end-users will systematically test is still needed.
  • Combining prioritization methods: The authors suggest exploring combinations of Confidence, Similarity, and Relevance methods to capture a wider range of bugs.
  • Broader applicability: While tested on text classification, the framework's potential for other types of intelligent assistants warrants further exploration.

Summary updated

Abstract

Intelligent assistants are handling increasingly critical tasks, but until now, end users have had no way to systematically assess where their assistants make mistakes. For some intelligent assistants, this is a serious problem: if the assistant is doing work that is important, such as assisting with qualitative research or monitoring an elderly parent’s safety, the user may pay a high cost for unnoticed mistakes. This paper addresses the problem with WYSIWYT/ML (What You See Is What You Test for Machine Learning), a human/computer partnership that enables end users to systematically test intelligent assistants. Our empirical evaluation shows that WYSIWYT/ML helped end users find assistants’ mistakes significantly more effectively than ad hoc testing. Not only did it allow users to assess an assistant’s work on an average of 117 predictions in only 10 minutes, it also scaled to a much larger data set, assessing an assistant’s work on 623 out of 1,448 predictions using only the users’ original 10 minutes’ testing effort.
BibTeX
@inproceedings{kulesza2011where,
  author = {Todd Kulesza and Margaret M. Burnett and Simone Stumpf and Weng-Keen Wong and Shubhomoy Das and Alex Groce and Amber Shinsel and Forrest Bice and Kevin McIntosh},
  title = {{Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach}},
  booktitle = {IS-EUD},
  year = {2011},
  url = {https://openreview.net/forum?id=uS0tWwM2sFz},
}
Figure