Supported What-Ifs: Image–Concept Contrast for Plausible Interactive Edits in Diagnostic Models

Interactive "what-if" tools let a clinician change a patient's attributes and watch a diagnostic model's prediction respond. When the model reasons through human-readable clinical concepts such as "spiculated margin" or "lobulated shape," these tools are a direct way to audit it. They have two quiet failure modes. First, editing one concept leaves every correlated concept frozen, so the simulated patient is often one who could not exist, for example a nodule made smooth but still marked as spiculated. Second, the tool answers every query with equal confidence, which lets users hunt for edits that confirm what they already believed. Liu et al. found that interactive tools felt more useful but did not raise decision accuracy, and that they led users to fixate on sensitive attributes (Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making).

We propose training a compatibility score between a patient's image and a full vector of clinical concepts, using contrastive learning. Contrastive learning here means teaching the model to score the true (image, concepts) pair above mismatched pairs. The score is used at interaction time for two jobs. When the user edits some concepts, it fills in the remaining concepts with the configuration most consistent with this patient's image. It also checks whether any real patient in the training data supports the edited configuration, and shows that patient as evidence or warns when none exists. Contrastive training is usually expensive because it needs several augmented copies of each image. We avoid that cost because the second "view" is the concept vector itself, so each image passes through the heavy backbone only once, and all extra negatives are generated in the cheap concept space.

Core Claim

A concept bottleneck model (a classifier forced to predict named concepts first and then the diagnosis from those concepts only, Concept Bottleneck Models) can be given an image-conditioned, nonlinear compatibility score $s(x,c) = f(x)^\top g(c)/\tau$. Here $f$ is the image backbone, $g$ is a small nonlinear network over the binary concept vector $c$, and $\tau$ is a temperature. The score is trained with an InfoNCE loss, the standard contrastive loss that treats one true pair and many mismatched pairs as a classification problem. The negatives are other patients' concept vectors in the batch, plus $K$ "counterfactual negatives" per image, made by flipping a few of the image's own concepts. The total cost is one backbone pass per image plus an $N \times (N+K)$ score matrix over low-dimensional vectors.

Given a user edit that fixes concepts $S$ to values $v$, the completed patient is

$$\hat c = \arg\max_{c:\ c_S = v} s(x, c),$$

and the edit's support is $\rho(c) = \max_{x' \in \mathcal{D}} s(x', c)$, calibrated against the score distribution of true pairs. Here $\mathcal{D}$ is the training set, whose image embeddings are cached. The argmax $x'$ is returned as the nearest supporting case.

We claim three things. These completions match held-out real patients better than leaving the other concepts fixed or filling them from population co-occurrence alone. Low-support flags predict which counterfactuals have no real-world realization. Training costs close to a plain concept bottleneck model rather than a two-view contrastive model. The nonlinearity of $g$ is essential. If $g$ is linear, $s$ splits into independent per-concept terms, and completion collapses to ordinary concept prediction with no propagation of correlations.

Positioning

The closest work is Counterfactual Concept Bottleneck Models (and its earlier version Climbing the Ladder of Interpretability with Counterfactual Concept Bottleneck Models). These papers add a variational counterfactual encoder that, given a target label $y'$, produces alternative concepts. They report 99% counterfactual validity on the CUB bird dataset, against 66% for the post-hoc search method BayCon. They supply our evaluation protocol and our strongest competitor, but they answer a different question. Their model asks "which concepts would flip the label?", and it is the model that chooses the edit. We ask "the user changed these concepts; what else follows for this patient, and has anything like it ever been seen?" They do not test whether a counterfactual is supported by real data.

Interactive Concept Bottleneck Models choose which concepts to ask a human about. They gain 5–10% accuracy within five interventions on CUB and CheXpert, but they assume the human is an error-free oracle and update only the concept that was queried. Our completion step is complementary: one human correction should also update the concepts correlated with it.

Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning uses contrastive and Gumbel-softmax objectives over a frozen CLIP image–text space and reaches 80.02% on CUB-200. They contrast images with concept text. We contrast images with structured concept vectors, adapt the backbone, and use the resulting score for interaction rather than only for classification.

Two works constrain the design. Counterfactual Data Augmentation with Contrastive Learning shows that a contrastive latent space finds outcome-relevant neighbours far better than propensity matching (error 0.83 vs 3.94 on IHDP). It also deliberately imputes counterfactuals only where dense neighbours exist. We carry that caution into interactive simulation as the support check. The overhead literature motivates the cheap second view. Two-branch contrast doubles forward passes and memory in speech and point-cloud models (Selective HuBERT: Self-Supervised Pre-Training for Target Speaker in Clean and Mixture Speech, What We Miss Matters: Learning from the Overlooked in Point Cloud Transformers), and CSE-SFP shows that getting two views from one pass can cut training time by 43% without losing quality. In addition, standard image augmentations can destroy diagnostic cues such as lesion margins. Using concepts as the second view sidesteps augmentation entirely.

Two works warn about labels. Integrating Clinically-Relevant Features into Skin Lesion Classification found that auxiliary clinical features hurt accuracy on benign nevi, so we must show that the contrastive term does not degrade the diagnosis. Re-thinking and Re-labeling LIDC-IDRI shows that radiologist malignancy scores are biased. We therefore evaluate lung-nodule counterfactuals against the biopsy-proven SCH-LND cohort, not only against consensus labels.

Research Program

The model is a standard concept bottleneck model: a backbone $f$, a concept head, and a sparse task head over concepts, plus the concept encoder $g$ (a two-layer network of about 0.1M parameters) and the compatibility loss. Counterfactual negatives flip between one and three concepts at a time. They are sampled from concepts the image's own prediction is uncertain about, so that the score learns fine distinctions where they matter.

At interaction time, completion is solved by beam search over the unedited binary concepts, warm-started from the factual prediction. Because $g$ is tiny and the 118 CUB concepts or 9 LIDC attributes are few, we target well under one second per query. That sits in the same regime as the 0.5–3.0 s that GAM Coach achieves for additive tabular models (GAM Coach), here for deep image models. Support is a single matrix product against the cached training embeddings. We will also study one variant: optionally restricting the user-facing prediction to supported completions. This lets us measure whether refusing or flagging changes how users probe the model.

Baselines

The naive reference is a plain concept bottleneck model with intervention semantics. The edited concepts are overwritten, all others keep their predicted values, and no support is reported. This is what current concept-level what-if tools do.

The strongest existing method is the Counterfactual Concept Bottleneck Model of Dominici et al. We adapt its counterfactual encoder to condition on the user's partial concept edit in place of, or alongside, the target label, retrain it on identical splits and backbones, and give it the same hyperparameter search budget and GPU hours. On its native task, label-flip validity on CUB, the paper reports 99%. Our model must not fall meaningfully below this when used for label-directed counterfactuals. On edit completion, we must beat the adapted version.

A purely data-driven baseline does k-nearest-neighbour completion in concept space. It fills unedited concepts from the training patients that best match the edit, and it measures support by neighbour distance. This uses no learning and no image.

A concept-only density baseline uses a small autoregressive model over concept vectors for completion and likelihood. It captures population co-occurrence but ignores the specific image.

Two ablations isolate the key ingredients. A linear $g$ tests whether interaction structure is what drives completion. A two-view image-augmentation contrastive model, trained alongside the concept bottleneck at the same batch size, tests whether concept-as-view saves compute without losing quality.

All methods share backbones (ResNet for CUB and CheXpert, a 3D CNN comparable in size to the 1.5M-parameter PN-SAMP for LIDC), data splits, and tuning budget. For the diagnostic task itself, PN-SAMP reports 97.58% malignancy accuracy and 89.33% attribute accuracy on LIDC-IDRI (Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction). Zhang et al. report sensitivity 0.67 and specificity 0.66 after relabeling.

Beating the baselines means two things. Completion accuracy must be higher than both the k-NN and density baselines, with confidence intervals that do not overlap across seeds. Task accuracy must stay within one point of the plain concept bottleneck model.

Why not just…?

Why not just use k-nearest neighbours in concept space for completion and support? This is the most serious threat. If concept correlations are mostly population-level, k-NN will match us. Our bet is that the right completion depends on the image: two patients with the same edit should get different filled-in concepts. The experiment that settles this stratifies results by how much the image-specific prediction differs from the population co-occurrence. If our model gains nothing over k-NN even in the high-disagreement strata, we will report that the learned score is unnecessary and that the support check should be built on k-NN.

Why not just model the joint distribution of concepts with a density model? A density model answers "is this configuration common?" but not "is it close to this patient?". A rare but genuine configuration that fits this patient's image would be wrongly flagged. The concept-only density baseline is run exactly to measure this gap. On LIDC we specifically test whether flags fall disproportionately on benign-but-irregular nodules. That is the kind of borderline case behind Veltmeijer et al.'s nevus result.

Why not just use the counterfactual encoder of the counterfactual concept bottleneck model, conditioned on edits? It can be adapted this way, and we do run it, but there are two concerns. Its variational sampling explores freely, with no requirement to stay near real data, and the authors themselves note it only approximates minimal changes. It also produces no evidence a user can inspect. If the adapted encoder matches our completion accuracy, the remaining difference is the support check and the nearest supporting case. We would then present those as a module that sits on top of their model.

Why not just use ordinary two-view contrastive pretraining? It doubles backbone passes, and on medical images its augmentations can blur exactly the margins and textures that concepts describe. The ablation reports wall-clock time, peak memory, and completion accuracy at matched batch size. We expect a large saving with no loss in accuracy, but the ablation will show whether that holds.

Proposed Experiments

On CUB, which has 118 annotated concepts, we evaluate completion as follows. We reveal a random subset of a held-out image's true concepts, ranging from 5% to 50%, and score accuracy on the remaining concepts against ground truth. We also rerun the Interactive Concept Bottleneck Models querying protocol, with and without completion, and plot task accuracy against the number of interventions relative to the gain of 5–10% within five that CooP reports.

For support, we construct edits whose target configurations either do or do not appear among held-out images of related species. The area under the ROC curve for separating the two groups measures whether flags track real-world realizability. On CheXpert and OAI we repeat the intervention-gain experiment. We found no reported completion or support numbers on Lacuna for any of these benchmarks, so baselines will be computed by us.

On LIDC-IDRI with the SCH-LND biopsy cohort, the decisive clinical check concerns counterfactuals that move a nodule toward "benign." Do those completions resemble biopsy-proven benign nodules more than the naive baseline's completions do, measured by embedding distance to SCH-LND benign cases? And do supported edits agree with pathology more often than unsupported ones? We will also measure task sensitivity and specificity, to confirm the contrastive term does not produce the auxiliary-feature distraction effect.

For efficiency, we report training time and memory against both the plain concept bottleneck model and the two-view ablation.

Finally, we run a small between-subjects study modelled on Liu et al., using COMPAS-style tabular concepts with a matching tabular encoder. Participants use the what-if tool either with or without support flags. We measure decision accuracy and how often they probe sensitive attributes. If flags change neither, we will report that the support check improves simulation fidelity but does not by itself correct human probing behaviour.