Overview based example selection in end user interactive concept learning

Saleema Amershi, James Fogarty, Ashish Kapoor, Desney S. TanUIST 2009 · DOI

TL;DR

This paper tackles the challenge of making interactive machine learning more intuitive for end-users. They introduce novel "overview-based" methods that visually summarize a machine learning model's current understanding of a concept, helping users select better training examples and train models more effectively. This research is crucial for building more user-friendly AI systems that can adapt to individual needs.

What Problem Does This Paper Address?

Imagine you're trying to find specific types of images on the web, say, "product photos of stereos." Traditional keyword search often falls short because keywords are too general or don't capture the visual nuances you're looking for. While machine learning can help, teaching a machine to recognize your specific concept interactively can be a frustrating experience. Users need to provide examples, but it's hard to know which examples are most helpful for the machine to learn. Existing methods often either overwhelm the user or don't provide enough guidance. This paper addresses this gap by exploring how to present the machine's current understanding of a concept in a way that empowers the user to make smarter training decisions. They focus on the challenge of guiding example selection in interactive concept learning, aiming to balance user control with effective machine learning.

Methods

The core innovation lies in their "overview-based" methods for presenting the machine's current concept definition. Instead of just showing raw results, they create visual summaries.

  • Semi-Supervised Classification: Before generating overviews, they first classify unlabeled images into "positive" and "negative" regions based on the current learned concept. They use a graph-based approach (k-nearest neighbors and geodesic distance) to ensure this classification respects the underlying data distribution, making it more robust than simple nearest-neighbor methods.
  • Global Overview: This method aims to provide a broad understanding of the entire concept space. It selects a set of representative images that maximize mutual information with the rest of the high-dimensional space, essentially picking images that best summarize the overall concept.
  • Projected Overview: This approach focuses on highlighting variations along key dimensions of the concept space. It identifies principal dimensions (similar to Principal Component Analysis but using geodesic distance) and then selects images that best illustrate variation along these dimensions while minimizing variation in others.
  • Neighbors: To provide context and aid interpretation, they also explore presenting selected overview images alongside their nearest neighbors. This helps users understand why a particular image was chosen and potentially label groups of similar images more efficiently.

These methods are implemented within a system called CueFlik, which allows users to interactively train machine learning models for image search.

Datasets

The paper uses image search queries and associated target images from a previous study [8] for evaluation. Specific queries mentioned include "stereo," "phone," "car," "character," "sky," "drink," "bill," "crowded," "disco," "tennis," and "sea." The system itself is trained on large, unstructured web image datasets associated with these queries. No new datasets were introduced.

Key Insights and Ideas

  • Visualizing the "Unknown" is Key: The most impactful insight is that showing users what the machine currently thinks (i.e., an overview of the positive and negative regions) is far more effective than just showing high-certainty examples. This visual feedback empowers users to make more informed decisions.
  • Overviews Reduce "Decay": Users often continue training even when a concept has reached its peak quality (BestScore). This "decay" can negatively impact the final model. Overview methods significantly reduce this decay, meaning users are less likely to over-train and degrade their concept.
  • Neighbors Can Be a Double-Edged Sword: While showing neighbors can help users interpret overviews and potentially speed up training, it surprisingly led to slightly worse final concept quality and slower training to reach the best concept. This suggests that focusing too much on local similarity might sometimes obscure the broader picture needed for optimal concept definition.
  • Variation is Good, But Needs Context: Both Global and Projected overviews, by emphasizing variation, led to better concepts. However, the feedback suggests that too much emphasis on variation can make it hard for users to judge when to stop training.

Key Results

The study compared their overview-based methods against a "HighCertainty" baseline (showing the most confident examples).

  • Better Concepts: Overview interfaces led to significantly better final concept scores (178 vs. 223 for HighCertainty) and better "best" concept scores (128 vs. 149 for HighCertainty).
  • More Efficient Training (to Best): While overall training time increased, users reached their best concept faster with overview methods (108 sec vs. 120 sec for HighCertainty).
  • Reduced Over-training: Overview interfaces resulted in less "decay" from the best concept to the final concept, indicating users were less likely to negatively impact their models by continuing to train.
  • NoNeighbors Outperforms WithNeighbors: Interfaces without neighbors generally yielded better final concept scores and better best concept scores, though they required slightly more time to reach the best concept.
  • Global vs. Projected: Global overviews led to participants spending less time training their final concept compared to Projected overviews.

Limitations and Future Work

  • User Frustration with Neighbors: Participants reported feeling more confused or frustrated with interfaces that included neighbors, and found them less easy to use. This suggests that while the idea of neighbors is good, the implementation or presentation needs refinement.
  • Difficulty in Judging Stopping Points: A recurring theme is the challenge users face in knowing when to stop training. While overviews help reduce the negative impact of over-training, actively guiding users on when to stop remains an open problem.
  • Understanding Decision Boundaries: The paper notes that even with overviews, understanding and manipulating the "decision boundary" (where the system switches from positive to negative) is crucial and interacts with guiding stopping points.
  • Future Directions: The authors suggest further research into:
  • Guiding users to select effective training examples in other applications.
  • Developing better methods for identifying effective stopping points.
  • Improving user understanding and manipulation of decision boundaries.
  • Exploring how different high-level user strategies (e.g., focusing on positive vs. negative examples) can be supported.

Summary updated

BibTeX
@inproceedings{amershi2009overview,
  author = {Saleema Amershi and James Fogarty and Ashish Kapoor and Desney S. Tan},
  title = {{Overview based example selection in end user interactive concept learning}},
  booktitle = {UIST},
  year = {2009},
  doi = {10.1145/1622176.1622222},
}
Figure