Towards Visual Explainable Active Learning for Zero-Shot Classification

Shichao Jia, Zeyu Li, Nuo Chen, Jiawan Zhang

TL;DR

The "Semantic Navigator" is a visual analytics system that transforms the tedious process of building zero-shot classification models into a collaborative "human-AI teaming" exercise. By using contrastive questions and a novel "semantic map," the system guides users to define the most impactful attributes, significantly improving both the efficiency of the labeling process and the final accuracy of the model.

What Problem Does This Paper Address?

Zero-Shot Classification (ZSC) is a powerful machine learning paradigm where a model learns to recognize objects it has never seen before (e.g., identifying a "zebra" after only being trained on "horses" and "stripes"). To do this, the model requires a class-attribute matrix—a structured map defining which classes possess which semantic traits.

Traditionally, creating this matrix is a nightmare for experts. It involves manually specifying dozens of attributes for hundreds of classes through trial and error. There is often no feedback on whether a chosen attribute (like "has ears") actually helps the model distinguish between classes, leading to a "semantic gap" where the machine's internal logic and the human's domain knowledge fail to align.

Methods

The authors propose a Visual Explainable Active Learning loop consisting of four iterative actions:

  1. Ask: Instead of asking broad questions, the machine asks contrastive questions (e.g., "What is the most visible difference between a deer and a bobcat?"). This leverages human analogical reasoning, which is more efficient than absolute description.
  2. Explain: The system provides a Semantic Map. This visualization projects the machine's high-dimensional "feature space" and the human's "semantic space" into a shared Mutual Mental Space. Using t-SNE, it shows class clusters (contours), where the machine thinks a class should be (prototypes), and where the actual data resides (exemplars).
  3. Recommend: To reduce the labeling burden, the machine uses a Semi-Supervised Support Vector Machine ($S^3VM$) to predict attribute labels for classes the user hasn't addressed yet.
  4. Respond: The user provides feedback by lasso-selecting classes on the map to confirm or correct the machine's recommendations.

The underlying ZSC model used is EXEM, chosen for its computational efficiency, which allows for the real-time updates necessary for an interactive system.

Datasets

The system was evaluated using two standard ZSC benchmarks:

  • AWA2 (Animals with Attributes 2): A dataset of 37,322 images across 50 mammal classes.
  • SUB-CUB200: A subset of the CUB-200-2011 bird dataset, featuring fine-grained bird species that are notoriously difficult to distinguish.

Key Insights and Ideas

  • Bridging the Semantic Gap: The "aha moment" of the paper is the Semantic Map. By visualizing the distance between a "prototype" (the attribute-based definition) and an "exemplar" (the actual image data), users can see exactly why a model is failing. If the prototype for "Wolf" is sitting inside the "Dog" cluster, the user knows they need an attribute to differentiate the two.
  • Semantic Trajectories: The system tracks how class prototypes move as new attributes are added. Users can watch a class "converge" toward its data cluster, providing immediate visual confirmation that their input is helping the model.
  • Contrastive Elicitation: The paper highlights that humans are much faster at identifying differences between two specific items than they are at listing all traits of a single item. The "Hint View" uses the model's own confusion matrix to pick the two classes the machine is most likely to mix up, forcing the human to provide the most "discriminative" information.

Key Results

  • Efficiency Gains: In user studies, participants using the Semantic Navigator saved an average of 14.42 seconds per attribute compared to those using a traditional manual matrix interface.
  • Accuracy Boost: Models built with the guided system saw a significant performance increase. On average, test accuracy improved by 4.96% for the AWA2 dataset and 4.13% for the SUB-CUB200 dataset compared to unguided baselines.
  • Qualitative Success: Users reported that the system made a "boring" and "exhausting" task feel intuitive. The visual feedback prevented the "brain stuck" feeling common in manual data labeling.

Limitations and Future Work

  • Binary Constraints: The current system only supports binary attributes (presence/absence). The authors suggest that future versions should support relative attributes (e.g., "Animal A is furrier than Animal B").
  • Feature Quality: The system assumes the underlying deep learning features are robust. If the raw image features are poor, no amount of attribute labeling can fix the model's performance.
  • Scalability: While it works well for 50 classes, the visual complexity of the Semantic Map might become overwhelming for datasets with hundreds or thousands of classes. Future work may need to incorporate hierarchical clustering or "focus+context" views.

Summary updated

Abstract

Zero-shot classification is a promising paradigm to solve an applicable problem when the training classes and test classes are disjoint. Achieving this usually needs experts to externalize their domain knowledge by manually specifying a class-attribute matrix to define which classes have which attributes. Designing a suitable class-attribute matrix is the key to the subsequent procedure, but this design process is tedious and trial-and-error with no guidance. This paper proposes a visual explainable active learning approach with its design and implementation called semantic navigator to solve the above problems. This approach promotes human-AI teaming with four actions (ask, explain, recommend, respond) in each interaction loop. The machine asks contrastive questions to guide humans in the thinking process of attributes. A novel visualization called semantic map explains the current status of the machine. Therefore analysts can better understand why the machine misclassifies objects. Moreover, the machine recommends the labels of classes for each attribute to ease the labeling burden. Finally, humans can steer the model by modifying the labels interactively, and the machine adjusts its recommendations. The visual explainable active learning approach improves humans' efficiency of building zero-shot classification models interactively, compared with the method without guidance. We justify our results with user studies using the standard benchmarks for zero-shot classification.
BibTeX
@article{jia2021towards,
  author = {Shichao Jia and Zeyu Li and Nuo Chen and Jiawan Zhang},
  title = {{Towards Visual Explainable Active Learning for Zero-Shot Classification}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2021},
  doi = {10.1109/tvcg.2021.3114793},
}

Loading related content...

Figure