Who Is Wrong: the Label, the Concept, or the Model? Spike-and-Slab Error Attribution for Concept Bottleneck Regression

A concept bottleneck model predicts in two steps. It first maps an input, such as a knee X-ray, to a handful of human-named attributes like "joint space narrowing." It then maps those attributes to the answer, such as a disease grade, with a simple linear rule that a clinician can read. That readable rule is the whole point of the design. It is fitted by least squares, though, and least squares lets a few corrupted training records pull the weights wherever they like. We expect contamination to damage the explanation, meaning which concepts get nonzero weight and with what sign, well before it visibly damages accuracy. If so, the interpretable model fails silently in the one respect it was built for.

We propose a single Bayesian model of the bottleneck-to-answer stage. It uses spike-and-slab priors, which are mixtures of an exact zero and a broad distribution, so each quantity is either "off" or "free to be large." These priors are placed on three sparse objects at once: the concept weights, a per-record shift on the label, and a per-record, per-concept shift on the annotated concepts. The posterior then states which concepts matter. For every suspicious training record it also states where the error most likely sits: in the label, in a specific concept annotation, or nowhere. Robustness and interpretability become the same output, a calibrated audit of the data written in the model's own vocabulary.

Core Claim

Robust regression and probabilistic variable selection have been combined before, but only for plain linear regression. That work also treats every outlier as a single scalar residual. High-dimensional outlier detection and variable selection via adaptive weighted mean regression fits a Huber loss with a Lasso penalty and per-sample weights, and proves that it detects outliers consistently. It motivates our work and also supplies our strongest baseline. Its sample weights cannot say why a record is bad.

A concept bottleneck offers something plain regression lacks: two independent measurements of every concept. One is the human annotation. The other is the image-based concept predictor, fitted on held-out folds so it cannot memorize annotation errors. The label gives a third, indirect constraint. With these three signals, an error in a concept annotation can be told apart from an error in the label. Because concept weights carry a spike-and-slab prior, the model can also tell when a disagreement does not matter, which happens when it falls on a concept whose weight is exactly zero.

Our claim has two parts. The first is that, under realistic contamination, a least-squares bottleneck head changes its selected concept set and weight signs substantially, even while held-out error moves little. The second is that the joint spike-and-slab model keeps the clean-data explanation and attributes errors by type and by concept more accurately than any combination of a robust loss, Lasso, and a disagreement threshold.

Model

For record $i$ and concept $j$, let $t_{ij}$ be the unobserved true concept value. The annotation is $a_{ij}$, the cross-fitted image prediction is $\hat c_{ij}$, and the response is $y_i$. The generative model is

$$a_{ij} = t_{ij} + \delta_{ij} + \epsilon_{ij}, \qquad \hat c_{ij} = t_{ij} + \eta_{ij}, \qquad y_i = \beta^\top t_i + \gamma_i + \xi_i,$$

Here $\epsilon,\eta,\xi$ are Gaussian noise terms. The concept weights $\beta_j$, label shifts $\gamma_i$, and concept-annotation shifts $\delta_{ij}$ each get a Dirac spike at zero mixed with a wide slab, with separate inclusion probabilities. Writing outliers as sparse additive shifts is the Bayesian counterpart of the sample weights in the Huber-Lasso work above. The Dirac spike gives exact zeros and posterior inclusion probabilities rather than thresholded magnitudes.

The head has only a few concepts and a few thousand records, and every conditional is Gaussian or Bernoulli. Exact collapsed Gibbs sampling is therefore feasible, so we do not need the continuous relaxations that Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee uses for full networks. That paper supplies the prior form and its evidence for clean selection. It also documents a sharp U-shaped sensitivity to the prior inclusion probability. We address that sensitivity with hyperpriors on the three inclusion rates instead of hand tuning.

Research Program

We first build the model on the sequential version of the bottleneck from Concept Bottleneck Models. We keep its image-to-concept network and replace only the least-squares head. Cross-fitting the concept predictor is essential, because otherwise $\hat c$ copies $a$ and annotation errors become invisible.

We then establish the motivating fact: how fragile the explanation is under contamination. We inject controlled label shifts and concept-annotation corruptions into OAI training data at rates of 0, 5, 10 and 20 percent. We track test error, the Jaccard overlap of the selected concept set with the clean-data fit, and the rate of weight-sign flips.

Next we evaluate attribution. Every injected corruption has a known type and location, so the model's posterior can be scored directly. We report detection precision and recall separately for label and concept errors. We report whether the flagged concept is the corrupted one, and whether nominal 90 percent inclusion probabilities are right about 90 percent of the time.

Finally we test a deployment use. Clinical labels often arrive late, for example at follow-up. When they do, the same posterior can tell a clinician which concept to re-read. We compare this targeted correction against the variance-based prioritization of Probabilistic Concept Bottleneck Models, which ranks concepts by perceptual ambiguity rather than by their inconsistency with the outcome.

Baselines

The naive reference is the original sequential and joint bottleneck with a least-squares head, as in Concept Bottleneck Models. On OAI that paper reports only that its RMSE is lower than a black-box CNN. No absolute RMSE for the bottleneck appears on Lacuna. The one number available is that correcting two concepts "reduced the task error from over 0.4 to approximately 0.3." We will therefore re-run this model ourselves and treat its clean-data RMSE as the target that the robust head must not lose by more than noise.

The strongest competitor is the penalized weighted Huber-Lasso of High-dimensional outlier detection and variable selection via adaptive weighted mean regression, fitted to the same predicted concepts. It already handles label outliers and selects variables, so it is the method most likely to make our proposal redundant. No reported result for its breast-cancer experiment was found on Lacuna; the paper describes it only as achieving the lowest MSE. We will run it on our splits, with the same cross-fitted concepts and the same budget for tuning its Huber threshold and penalty on a clean validation fold. We beat it if we match its clean-test RMSE and exceed it on two things: explanation stability at 10 to 20 percent contamination, and concept-level attribution, which it cannot produce natively.

To give it a fair attribution score, we pair it with a strong add-on: flag concept $j$ of record $i$ when the annotation-minus-prediction gap exceeds a threshold tuned on validation data. We also run a plain mean-absolute-error head, following On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression. For selection quality, the reference points come from Sparsifying Bayesian neural networks with latent binary variables and normalizing flows, which reports a true positive rate of 0.972 for its flow model against 0.681 for Carbonetto-Stephens under correlated designs. We use these only to calibrate expectations on our synthetic suite.

Three ablations isolate the ingredients. The first drops the concept shifts $\delta$, leaving a Bayesian version of label-only outlier modeling. The second swaps the Dirac spike for a Laplace prior, a Bayesian Lasso, to test whether exact zeros matter. The third removes cross-fitting to show that the second concept measurement is what makes errors identifiable.

Why not just…?

Why not just use Huber-Lasso on the concept head? For label outliers alone it may be as good, and we take that threat seriously. If it matches the full model on label detection and explanation stability, the contribution shrinks to concept-level attribution, and the label-only ablation will show this directly. Its single sample weight per record still cannot distinguish a wrong grade from a wrong "bone spur" annotation. Downweighting a record whose label is correct but whose concept annotation is wrong discards good information about the label.

Why not just flag concepts where the annotation disagrees with the image predictor? This is the most dangerous simple alternative for the attribution claim. Its blind spot is predictable. It flags disagreements on concepts with zero weight, which do not affect the answer. It also cannot use the label to decide which of the two concept measurements is wrong. The thresholded-disagreement baseline above tests exactly this. If it matches our concept-attribution precision at equal recall, we will report that the Bayesian machinery is unnecessary for this part.

Why not just use ProbCBM's concept variance? That variance measures how ambiguous the image is, not whether the annotation contradicts the outcome. A clear image with a mistyped annotation has low variance. Our deployment experiment compares the two rankings head to head on records with known injected errors.

Why not just train with mean absolute error? It limits the pull of large residuals. As the Huber-Lasso work argues, though, unweighted losses still suffer masking, where clusters of outliers hide one another, and they offer no record-level diagnosis. It is included as a baseline, and the stability metric will show whether it suffices.

Proposed Experiments

We use three settings. The first is a synthetic sparse linear suite with correlated concepts and known corruptions, used to check identifiability and calibration. The second is OAI knee-grade regression with injected contamination, our main testbed, chosen because it is the only regression benchmark in this literature with clinical concept annotations. The third is the breast-cancer tumor-diameter data used by the Huber-Lasso work. It has no concepts, so it tests only the label-shift and weight components, on naturally occurring outliers, against that method on its own ground.

The decisive checks are three. First, the least-squares head's concept-set overlap with the clean fit must fall noticeably at 10 percent contamination while its RMSE barely moves. If the explanation is not fragile, the motivation weakens. Second, the full model must beat the Huber-Lasso-plus-threshold baseline on concept-level attribution F1 at matched clean RMSE. Third, posterior inclusion probabilities for both outlier types must be calibrated on the synthetic suite. If the second check fails, we will report the full model as an expensive way to reproduce simple diagnostics.