Selective Prediction For Open-Ended Question Answering in Black-Box Vision-Language Models

Zaid Khan, Yun FuR0-FoMo @ NeurIPS 2023 · OpenReview

TL;DR

This paper introduces a novel method for "selective prediction" in black-box vision-language models (VLMs), allowing them to abstain from answering when uncertain. By measuring the consistency of a VLM's answers across different phrasings of a question, the authors demonstrate a reliable way to identify high-risk queries, even for models accessed only via API. This work is crucial for deploying powerful VLMs safely in sensitive applications.

What Problem Does This Paper Address?

Deploying advanced AI models, especially large vision-language models (VLMs), in real-world applications often involves significant risks. If a VLM makes a mistake in a critical scenario (e.g., medical diagnosis, autonomous driving), the consequences can be severe. This necessitates a mechanism for the model to know when it doesn't know, a concept known as "selective prediction" or "abstention."

Traditional methods for achieving selective prediction often require deep access to the model's internals (like gradients or feature representations), the ability to retrain the model, or a large number of evaluations. These requirements are incompatible with modern "black-box" models, which are often accessed through APIs for commercial, privacy, or security reasons. This creates a significant barrier to using these powerful, pre-trained foundation models in high-stakes applications. Furthermore, much of the existing work has focused on simpler, unimodal tasks (like image classification) rather than the more complex, open-ended nature of visual question answering (VQA).

Methods

The core idea of this paper is to leverage the principle of consistency over neighborhood sampling for uncertainty estimation, adapted for black-box VLMs. In white-box settings, this involves perturbing input features and observing how the model's prediction changes. Since direct feature access is impossible here, the authors propose using rephrased questions as a proxy for "neighborhood samples."

Their approach involves:

  1. Probing Model (VQG): They use a separate, smaller Visual Question Generation (VQG) model. This VQG model is trained to approximate the conditional probability of generating a question given an image and an answer ($p(Q|A, V)$).
  2. Generating Rephrasings: For a given image-question pair $(v, q)$, they first get an answer $a$ from the black-box VLM $f_{VQA}$. Then, they feed $v$ and $a$ into the VQG model to generate multiple rephrased questions $q'$.
  3. Measuring Consistency: The black-box VLM $f_{VQA}$ is then queried with the original image $v$ and each of the generated rephrased questions $q'$. The consistency of $f_{VQA}$ is measured by how many of these rephrased questions yield the same answer as the original question.
  4. Selective Prediction: A higher degree of consistency across rephrasings suggests the VLM's understanding is more robust and less sensitive to superficial variations in the question's wording. Conversely, low consistency indicates potential unreliability, prompting the model to abstain.

This method is "model-agnostic" as it doesn't rely on the internal architecture of the black-box VLM, and it avoids the need for retraining or large validation sets.

Datasets

The experiments were conducted on three types of datasets:

  • In-distribution: VQAv2 validation set.
  • Out-of-distribution (OOD): OK-VQA, which requires external knowledge not typically found in VLMs' pretraining data.
  • Adversarial: AdVQA, which contains questions specifically designed to challenge VQA models.

The VQG model was trained on the VQAv2 training set.

Key Insights and Ideas

  • Rephrasing as a Proxy for Neighborhood Sampling: The paper cleverly adapts the concept of exploring a model's "neighborhood" in feature space to the black-box setting by using linguistic variations (rephrased questions) instead of feature perturbations.
  • Consistency as a Reliable Uncertainty Signal: The core insight is that a VLM's consistency in answering across different phrasings of the same question is a strong indicator of its reliability. If a model's answer changes drastically with minor wording changes, it suggests a superficial understanding.
  • Black-Box Applicability: The method's independence from model internals makes it directly applicable to commercial APIs, opening doors for safer deployment of powerful VLMs.
  • Beyond Confidence Scores: The paper shows that traditional confidence scores alone are insufficient, especially for OOD and adversarial questions, where correct answers can have low confidence and incorrect answers high confidence. Consistency provides a complementary signal.

Key Results

  • Correlation with Accuracy: The study found a strong correlation between answer consistency and accuracy across all three datasets (in-distribution, OOD, and adversarial).
  • Risk-Coverage Tradeoff: By stratifying questions based on consistency levels, the authors demonstrated that higher consistency slices of the dataset allowed for achieving lower risk (fewer incorrect answers) at equivalent coverage (percentage of data processed). This held true even when the VQG model was significantly smaller than the black-box VLM.
  • Effective Rejection: The proposed method effectively identifies high-risk questions, enabling selective prediction to reject them and thus improve the overall reliability of the VQA system. For instance, on adversarial questions, higher consistency levels significantly reduced the risk of incorrect answers.

Limitations and Future Work

  • "Pseudo-Rephrasings": The generated rephrasings are not always perfect linguistic rephrasings; they are sometimes "pseudo-rephrasings." However, the paper argues they are sufficient for the consistency measure to be effective.
  • Hallucinations: The VQG model can sometimes "hallucinate" questions that assume facts not present in the image, especially when given out-of-context answers. This highlights a limitation in the VQG model's understanding of counterfactuals.
  • Calibration: While consistency helps, the paper notes that even with temperature scaling, models can remain overconfident on adversarial samples, suggesting calibration alone is not sufficient for selective prediction.
  • Future Work: The authors suggest this work is a step towards safer VLM usage and implies further research into improving VQG models and exploring other consistency-based methods for black-box uncertainty estimation.

Summary updated

Abstract

When mistakes have serious consequences, reliable use of a model requires understanding when the predictions of the model are trustworthy. One approach is selective prediction, in which a model is allowed to abstain if it is uncertain. Existing methods for selective prediction require access to model internals, retraining, or large number of model evaluations, and cannot be used for black box models available only through an API. This is a barrier to the use of powerful commercial foundation models in risk-sensitive applications. Furthermore, existing work has largely focused on unimodal foundation models. We propose a method to improve selective prediction in a black box vision-language model by measuring consistency over the neighbors of a visual question. Although direct sampling of the neighborhood is not possible, we propose using a probing model as a proxy. We describe experiments testing the proposed method on in-distribution, out-of-distribution and adversarial questions. We find that the consistency of a vision-language model across rephrasings of a visual question can be used to identify and reject high-risk visual questions, even in out-of-distribution and adversarial settings, constituting a step towards safe use of black-box vision-language models.
BibTeX
@inproceedings{khan2023selective,
  author = {Zaid Khan and Yun Fu},
  title = {{Selective Prediction For Open-Ended Question Answering in Black-Box Vision-Language Models}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=7GHPcloiHq},
}
Figure