Abstract
Given a (machine learning) classifier and a collection of unlabeled data, how can we efficiently identify misclassification patterns presented in this dataset? To address this problem, we propose a human-machine collaborative framework that consists of a team of human annotators and a sequential recommendation algorithm. The recommendation algorithm is conceptualized as a stochastic sampler that, in each round, queries the annotators a subset of samples for their true labels and obtains the information on whether the samples are misclassified. The sampling mechanism needs to balance between discovering new patterns of misclassification (exploration) and confirming the potential patterns of classification (exploitation). We construct a determinantal point process, whose intensity balances the exploration-exploitation trade-off through the weighted update of the posterior at each round to form the generator of the stochastic sampler. The numerical results empirically demonstrate the competitive performance of our framework on multiple dataset at various signal-to-noise ratios.
Introduction
Machine learning models often fail in systematic, predictable ways—such as an object detector consistently missing pedestrians in low-light conditions. When AI models are deployed in the real world, they frequently encounter data distributions that differ from their training sets, leading to performance degradation. These failures are often not random but cluster around specific attributes, termed failure patterns.
Identifying these patterns is crucial for safety and model debugging. However, determining if a model has failed typically requires ground-truth labels. In massive unlabeled datasets, labeling every instance is prohibitively expensive. This research addresses the challenge of designing a "smart sampler" that queries human annotators for labels of a minimal subset of samples to identify as many systematic failure clusters as possible. We introduce a human-machine collaborative framework that utilizes Gaussian Processes and Determinantal Point Processes to hunt down these failure patterns significantly faster and more reliably than standard active learning or random sampling.
Methodology
The proposed human-machine collaborative framework consists of two primary mathematical engines designed to optimize the search for failure patterns.
Gaussian Process for Value-of-Interest (VoI)
The system maintains a "belief" about which regions of the data space are likely to contain misclassifications. It employs a Gaussian Process (GP) to model the probability that a sample is part of a failure pattern. This belief is updated iteratively as human annotators provide labels, refining the model's understanding of the failure surface.
Determinantal Point Process (DPP) Sampler
To select the next batch of samples for human labeling, the framework uses a Determinantal Point Process (DPP). This approach elegantly balances two competing objectives:
- Exploitation: Querying samples near known misclassifications to "fill in" and confirm a suspected failure pattern.
- Exploration: Querying diverse, distant samples to ensure the system does not overlook entirely new failure modes.
A parameter ($\vartheta$) allows users to tune this balance. Empirical findings suggest that a slight lean toward exploitation ($\vartheta = 0.25$) is often the "sweet spot" for identifying patterns in noisy data.
Feature-Pseudolabel Kernel
A novel aspect of this approach is the similarity measure used. Beyond raw data features, the framework incorporates the model's "pseudolabels" (predicted classes). If two samples share similar features and the model assigns them the same (potentially incorrect) prediction, they are considered more likely to belong to the same failure pattern.
Experimental Evaluation
The framework was evaluated on 15 real-world datasets adapted from the "Domino" benchmark. These datasets represent various domains and are categorized by their Signal-to-Noise Ratio (SNR):
- Low SNR: Characterized by many random misclassifications (noise), making systematic patterns difficult to isolate.
- Medium/High SNR: Characterized by clearer, more distinct failure clusters.
The proposed method, dubbed Directed Sampling (DS), was compared against Uniform Sampling (US) and state-of-the-art Active Learning methods (e.g., BADGE, Coreset).
Key Results and Discussion
The experimental results highlight several critical insights:
- Active Learning vs. Failure Identification: Standard active learning aims to improve model accuracy through retraining. This study demonstrates that active learning is not the optimal tool for finding failures in a fixed, deployed model. Popular active learning methods performed poorly at this specific task compared to Directed Sampling.
- Sensitivity and Speed: DS identified the first failure pattern much faster than baseline methods. In many scenarios, it required querying only 11% of the dataset to find the first pattern, whereas uniform sampling required 33%.
- Effectiveness: With a query budget of 20% of the dataset, DS variants identified between 33% and 52% of all existing failure patterns. In contrast, uniform sampling and active learning methods identified fewer than 6%.
- Robustness: Even in Low SNR environments, the framework's exploration component prevented it from becoming trapped in noisy regions that did not constitute systematic patterns.
Limitations
The method's effectiveness is dependent on the quality of the data's feature representation (embeddings). If features do not capture the attributes causing model failure, the GP cannot effectively model the failure surface. Additionally, while greedy heuristics are used for DPP sampling, computational complexity remains a consideration for datasets with millions of points. Finally, the framework currently assumes human annotators act as perfect oracles; future work could address noisy feedback or annotator disagreements.
Conclusion
This research presents a robust framework for the efficient identification of systematic failures in predictive algorithms. By combining Gaussian Processes with Determinantal Point Processes, the Directed Sampling approach provides a scalable and effective solution for model debugging and safety assurance in real-world AI deployments.
Summary updated