VALIANT: Prompt Instability for Active Learning in Black-Box Medical Imaging

Dwarikanath Mahapatra, Behzad Bozorgtabar, Sudipta Roy, Imran Razzak, Mauricio Reyes

Foundation models (FMs) are transforming medical imaging, but they come with a catch: they are often "black boxes." Whether due to proprietary API restrictions or massive computational requirements, researchers frequently cannot access the internal weights or gradients of these models. This makes Active Learning (AL)—the process of strategically choosing which data points to label to save costs—notoriously difficult.

A new paper from the 40th AAAI Conference, VALIANT, introduces a clever way to perform active learning on these locked-down models by measuring "prompt instability."

TL;DR

VALIANT is a novel active learning framework that adapts black-box medical foundation models without needing access to their internal parameters. By using a lightweight "Visual Prompt Decoder" and a new metric called prompt instability, it identifies the most informative images to label, matching or exceeding full-supervision performance while using significantly less data.

What Problem Does This Paper Address?

In medical AI, labeling data is the primary bottleneck; you need expensive expert time (radiologists, pathologists) to annotate images. Active Learning (AL) tries to solve this by picking only the most "informative" samples for humans to label.

However, most state-of-the-art AL methods require "white-box" access—they need to see the model’s gradients or internal uncertainty. When a model is hosted behind an API (like a black box), these methods fail. Furthermore, standard fine-tuning is often too heavy for medical clinics. VALIANT tackles the challenge of:

  1. Label Scarcity: Reducing the number of expert annotations needed.
  2. Black-Box Constraints: Adapting models where internal weights and gradients are inaccessible.
  3. Domain Specificity: Ensuring visual prompts are grounded in actual medical anatomy rather than just random pixel noise.

Methods

The authors introduce VALIANT (Visual Adaptation and Learning Integration for Active learNing Tasks), which consists of three main pillars:

  1. Visual Prompt Decoder (VIPD): Instead of tuning the foundation model, VALIANT trains a tiny side-network (the VIPD). This network generates a "visual prompt"—a small modification to the input image—that helps the black-box FM understand the specific medical task.
  2. Zero-Order Optimization (ZOO): Since the FM's gradients are unknown, the authors use SPSA-GC (Simultaneous Perturbation Stochastic Approximation with Gradient Correction). This allows them to estimate how to improve the VIPD by only looking at the FM's final outputs (predictions), not its internals.
  3. Prompt Instability Metric: This is the core innovation for data selection. To decide which image to label, VALIANT slightly perturbs an image and sees how much the generated visual prompt changes. If the prompt is "unstable" (changes significantly), it suggests the image lies near a decision boundary and is highly informative for the model.
  4. Anatomical Grounding: The VIPD incorporates unsupervised segmentation maps. This ensures the prompts focus on relevant biological structures (like the lungs in a chest X-ray) rather than irrelevant background pixels.

Datasets

The researchers evaluated VALIANT across four diverse medical imaging benchmarks:

  • CheXpert: Chest X-rays for multi-label classification.
  • PCam (PatchCamelyon): Histopathology slides for breast cancer detection.
  • ISIC 2019: Dermoscopic images for skin lesion classification.
  • REFUGE: Retinal fundus images for glaucoma assessment.

Key Insights and Ideas

  • Instability as a Proxy for Uncertainty: The most brilliant move here is using the sensitivity of the prompt as a selection criteria. If a tiny nudge to the input makes the VIPD "rethink" its entire strategy for that image, that image is a prime candidate for human labeling.
  • Anatomy prevents "Adversarial" Prompts: Without the anatomical coherence loss ($L_{coh}$), visual prompts can become "noisy" or adversarial. By forcing the prompts to align with unsupervised segmentation masks, the model stays focused on medically relevant features.
  • Multi-Modality Efficiency: The authors found that a single VIPD trained on a mixture of modalities (VALIANT-Multi) often performed better than training separate decoders for each task, suggesting the framework learns a generalized "medical intuition."

Key Results

VALIANT consistently outperformed standard AL baselines like Random Sampling, Entropy-based uncertainty, and Coreset.

  • Efficiency: On the CheXpert dataset, VALIANT reached an AUC of 92.6% using only 50% of the training data. For context, a fully supervised model trained on 100% of the data only reached 92.4%.
  • Black-Box Superiority: It significantly outperformed BAPS, the previous state-of-the-art for black-box prompt optimization.
  • Foundation Model Agnostic: The framework showed gains regardless of whether the underlying FM was BioVil or MedCLIP, proving its versatility.

Limitations and Future Work

While impressive, the authors acknowledge a few hurdles:

  • Computational Cost of ZOO: Zero-order optimization requires many forward passes through the foundation model to estimate a single gradient step. While it saves on "labeling cost," it increases "inference cost" during training.
  • Task Scope: The current work focuses on classification. The authors suggest that extending this "prompt instability" logic to more complex tasks like image segmentation or detection is a promising next step.
  • Optimization Strategies: Future work will look into more efficient gradient-free optimization techniques to speed up the training of the VIPD.

Summary updated

Abstract

The deployment of large, black-box foundation models for medical image classification is often hindered by the high cost of acquiring large, task-specific labeled datasets for fine-tuning. While active learning (AL) presents a promising solution, many state-of-the-art AL methods are computationally expensive or require full access to internal model parameters. We present VALIANT (Visual Adaptation and Learning Integration for Active learNing Tasks), a new active learning framework designed to efficiently adapt black-box foundation models by overcoming these limitations. VALIANT introduces a lightweight Visual Prompt Decoder (VIPD), trained via unsupervised Zero-Order Optimization (ZOO), to generate task-specific visual prompts without internal model access. Our core contribution is a perturbation-based ranking strategy that leverages this VIPD to formulate a computationally efficient, gradient-aware informativeness metric. This metric, which we term prompt instability, identifies the most impactful samples for the labeling budget. VALIANT further enhances this process by incorporating anatomical information from unsupervised segmentation maps to generate more discriminative visual prompts. Extensive evaluations on multiple medical datasets demonstrate VALIANT’s superior performance and significant reduction in labeling costs compared to a range of existing active learning techniques, positioning it as a scalable and practical solution for medical image analysis.
BibTeX
@inproceedings{mahapatra2026valiant,
  author = {Dwarikanath Mahapatra and Behzad Bozorgtabar and Sudipta Roy and Imran Razzak and Mauricio Reyes},
  title = {{VALIANT: Prompt Instability for Active Learning in Black-Box Medical Imaging}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i10.37734},
}
Figure