Active Bayesian Assessment for Black-Box Classifiers

Disi Ji, Robert Logan, Padhraic Smyth, Mark SteyversZenodo 2020 · DOI · OpenAlex

Introduction

As machine learning models move from research labs to critical real-world applications—like medical diagnosis or autonomous driving—the need to evaluate them rigorously has never been higher. However, we often face a "black-box" problem: we have the model's predictions, but no access to its internal weights or training data. Worse, evaluating these models in a new environment requires human-labeled data, which is expensive and slow to collect.

In their paper "Active Bayesian Assessment of Black-Box Classifiers," Disi Ji and colleagues from UC Irvine propose a framework that combines Bayesian inference with active learning. They demonstrate how to assess a model’s accuracy, calibration, and flaws using significantly fewer labels than traditional random sampling.

The Challenge of Black-Box Assessment

The standard way to test an AI model is "passive" assessment: taking a random sample of data, obtaining human labels, and calculating a score like accuracy. This is inefficient because not all data points are equally informative. For example, if a model is 99% confident and correct on 1,000 "cat" images, labeling the 1,001st "cat" provides minimal new information.

Figure 1: Performance metrics like Accuracy and Expected Calibration Error (ECE) vary significantly across different classes, highlighting the need for targeted assessment.

Users of black-box models typically need to address three core tasks:

  1. Estimation: Determining the model's accuracy on specific target data.
  2. Identification: Pinpointing which classes the model fails on most frequently.
  3. Comparison: Assessing relative performance across different categories (e.g., "trees" vs. "humans").

Active Bayesian Framework

The authors propose an Active Bayesian Assessment framework. Instead of treating metrics as static values, they maintain a posterior distribution—a range of likely values—for every metric.

Methodology

  1. Bayesian Inference: The framework uses conjugate priors (e.g., Beta distributions for accuracy, Dirichlet for confusion matrices) to quantify uncertainty. As labels are acquired, the uncertainty "cloud" shrinks.
  2. Thompson Sampling: Assessment is framed as a "multi-armed bandit" problem where each "arm" represents a data group (e.g., a predicted class).
  3. Task-Specific Rewards: The algorithm selects groups to sample based on a reward function. For estimation, it targets groups with high variance to reduce overall uncertainty. For identification, it targets groups likely to be the "worst performers."
  4. Informative Priors: A key innovation is using the model’s own confidence scores to set an initial "guess" (prior). Even if the model is poorly calibrated, these scores provide a roadmap for where errors are likely to occur.

Key Results and Efficiency Gains

The researchers evaluated their framework using state-of-the-art models (ResNet, BERT) across diverse datasets including CIFAR-100, ImageNet, SVHN, 20 Newsgroups, and DBpedia.

The results demonstrate massive gains in label efficiency. For instance, to identify the least accurate class on ImageNet (1,000 classes), the active method required only 9.3% of the labels needed by random sampling to reach the same certainty level.

Figure 6: Comparison of identification performance (MRR) for top-1 and top-10 worst classes. The Active Bayesian approach (IPrior+TS) significantly outperforms uniform (UPrior) and passive (IPrior) baselines.

Insights

  • Uncertainty as a Tool: Modeling what is unknown allows for targeting labeling efforts where they are most impactful.
  • Self-Informing Models: A model's confidence scores, even if imperfect, significantly speed up assessment compared to starting with no information.
  • Scalability: The method is particularly effective for "extreme" tasks, such as finding needles in the haystack of high-dimensional datasets like ImageNet.

Limitations and Future Directions

While powerful, the current framework has areas for improvement:

  • Independence Assumption: It currently assumes performance in one class (e.g., "dogs") is independent of another (e.g., "cats"), ignoring potential hierarchical relationships.
  • Metric Continuity: The reliance on "binning" confidence scores could be replaced by continuous functions for smoother calibration modeling.
  • Dynamic Environments: Adapting the framework to "drifting" environments where performance changes over time remains a promising next step.

By transforming model evaluation into an active search for knowledge, this work provides a practical toolkit for auditing complex AI systems under tight budget constraints.

Summary updated

Abstract

To assess performance characteristics of neural models on several standard image and text classification datasets, we create assessment datasets based on output from the correspnding predicton models on standard test sets used for each dataset in the literature. The image datasets we use are: CIFAR-100 (Krizhevsky and Hinton, 2009), SVHN (Netzer et al., 2011) and ImageNet (Russakovsky et al., 2015). The text datasets we use are: 20 Newsgroups (Lang, 1995) and DBpedia (Zhang et al., 2015). For image classification we use ResNet (He et al., 2016) architectures with either 110 layers (CIFAR-100) or 152 layers (SVHN and ImageNet). For ImageNet we use the pretrained model provided by PyTorch, and for CIFAR and SVHN we use the pretrained model checkpoints provided at: https://github.com/bearpaw/pytorch-classification. For text classification tasks we use fine-tuned BERTBASE (Devlin et al., 2019) models. These models were all trained on standard training sets in the literature, independent from the datasets used for assessment.
BibTeX
@inproceedings{ji2020active,
  author = {Disi Ji and Robert Logan and Padhraic Smyth and Mark Steyvers},
  title = {{Active Bayesian Assessment for Black-Box Classifiers}},
  booktitle = {Zenodo},
  year = {2020},
  doi = {10.5281/zenodo.3672105},
}

Loading related content...

Figure