Keeping Concept Detectors Where They Belong: Cheap Jacobian Locality Penalties for Concept Bottleneck Models

A concept bottleneck model is an image classifier that first predicts human-readable attributes (such as "red wing" or "hooked beak") and then predicts the label from those attributes alone. Its main selling point is that a person can inspect the attributes and correct a wrong one at test time. That selling point fails if the attribute detectors are looking in the wrong place. On the Concept Trustworthiness in Concept Bottleneck Models found that vanilla concept models on the CUB birds benchmark ground their concepts in the correct image region only 11.7% to 26.2% of the time. For example, a model may predict "beak shape" by looking at the feet. When the "wing color" detector can see the whole bird, it can also carry species information that the label predictor then exploits. Addressing Leakage in Concept Bottleneck Models showed that this kind of leakage makes human interventions unreliable.

We propose to train concept detectors that are provably insensitive, to first order, to pixels outside the body part their concept names. Automatic differentiation computes $\partial c_k/\partial x$, the sensitivity of concept $k$ to each input pixel. We penalize the part of that sensitivity that falls outside a region derived from CUB's existing part keypoints, and we backpropagate through the penalty itself. The obvious version needs one extra backward pass per concept, which is 112 on CUB and too expensive. Our technical contribution is an unbiased random-projection estimator. Concepts that share a body part also share a mask, so we can combine them with random signs. This cuts the cost to roughly one extra backward pass per body part, or fewer, while keeping the expected penalty exact. The empirical claim is twofold. Locality enforced this way should close most of the trustworthiness gap without the part-prototype machinery of ProtoCBM. It should also reduce leakage as a side effect, because a detector that cannot see the rest of the bird cannot smuggle species identity into its output.

Core Claim

Three claims define the contribution, and each is individually testable. First, the attribute names in CUB already contain free spatial supervision. Each of the 112 attributes used by Concept Bottleneck Models names a body part (wing, beak, crown, and so on), and CUB annotates part keypoints per image. The attribute-to-part mapping therefore costs one afternoon of work rather than new annotation. Second, penalizing the concept Jacobian outside the part region during training yields concept detectors whose locality holds up under a different test: occlusion, which is not gradient-based. Third, this locality constraint moves the accuracy–interpretability frontier. At matched task accuracy it improves trustworthiness, and it improves the monotonicity of intervention curves.

Let $c_k(x)$ be the logit of concept $k$ on image $x$. Let $p(k)$ be the body part that concept $k$ names, and let $M_{p}(x)$ be a binary pixel mask equal to 1 outside a disc of radius $r$ around part $p$'s keypoint. The penalty is

$$R(x) = \sum_{k} \big\| M_{p(k)}(x) \odot \nabla_x c_k(x) \big\|_2^2 .$$

Here $\odot$ denotes pixelwise multiplication. For a group $G_p$ of concepts sharing part $p$, draw independent random signs $s_k \in \{\pm 1\}$. Compute a single vector–Jacobian product $v_p = \nabla_x \sum_{k \in G_p} s_k c_k$. Because the mask is shared within the group, the cross terms vanish in expectation:

$$\mathbb{E}_s \big\| M_p \odot v_p \big\|_2^2 = \sum_{k \in G_p} \big\| M_p \odot \nabla_x c_k \big\|_2^2 .$$

The penalty therefore needs one backward pass per part, and training through it needs one second-order (double-backward) pass per part. If we sample a random subset of parts per batch, the cost drops further. CUB has 15 part keypoints, so the worst case is about 15 extra passes instead of 112. Sampling two or three parts per batch brings the overhead to a small multiple of standard training. We will also apply the penalty on the backbone's spatial feature map instead of on pixels. This is cheaper, and pixel gradients are known to be noisy. Whether pixel or feature placement grounds concepts better is an empirical question that the ablations answer.

Relation to Prior Work

The motivating evidence comes from two papers. Huang et al. (On the Concept Trustworthiness in Concept Bottleneck Models) measured the grounding failure and fixed it architecturally. ProtoCBM uses learnable part-prototypes, cross-layer alignment, cross-image alignment and a prediction-alignment loss, and it reaches 62.8% to 70.5% trustworthiness. It deliberately avoids spatial supervision during training. We take the opposite bet. We use cheap, already-available keypoint supervision to shape an unmodified concept model, with no new architecture. If that matches ProtoCBM, locality becomes a single loss term that can be added to any bottleneck model. Chauhan et al. (Addressing Leakage in Concept Bottleneck Models) formalized leakage and attacked it with hard concepts, an autoregressive concept predictor and a latent side channel. We attack one source of leakage: concept detectors see non-concept evidence in the first place. The two approaches are complementary, and we test them combined.

Several papers constrain the design. Concepts' Information Bottleneck Models reduces leakage by penalizing mutual information between input and concepts. It does not evaluate spatial grounding, and it relies on a tuned $\beta$ plus a Gaussian assumption. Its penalty is global and ours is spatial, so the comparison tells us whether where information enters matters beyond how much. Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models is the closest methodological template. It adds a differentiable interpretability penalty to the training loss and traces the trade-off curve with one weight. It reduced concurvity from about 0.2 to 0.05 on California Housing while RMSE rose from 0.59 to 0.66. Its setting is tabular inputs, however, and its penalty acts on shape-function outputs rather than on input sensitivities of a deep feature extractor. Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis also uses spatial masks to guide concept alignment. It aligns latent axes by rotation in a decoupled two-branch procedure, whereas we constrain input sensitivity end to end.

On the automatic differentiation side, the method borrows a lesson from A Versatile Influence Function for Data Attribution with Non-Decomposable Loss. That paper showed that generic reverse-mode derivatives and second-order products, rather than hand derivation, make previously awkward sensitivity quantities practical. We apply that lesson to a training-time penalty rather than a post-hoc attribution. Concept Influence computes gradients with respect to concept directions on frozen models to attribute behavior to training data, and it never changes the model. We use concept-level gradients to change the model. One honest caveat is needed: penalizing input gradients is a known general idea. What is new here is (a) the free supervision derived from part-named concepts, (b) the shared-mask random-projection estimator that makes a per-concept penalty affordable, and (c) the test of whether gradient locality translates into occlusion locality, leakage reduction and better interventions in bottleneck models.

Research Program

We will first build the attribute-to-part map for CUB's 112 concepts. Attributes naming whole-body properties such as "size" or "shape" get no mask and serve as an internal control. We will then implement the penalty in PyTorch and JAX for a joint concept model on a ResNet18 backbone, matching the backbone Huang et al. used. We will validate the estimator by checking its variance and bias against the exact 112-pass penalty on a subset of batches. The main study sweeps the penalty weight $\lambda$ and radius $r$ to trace the frontier between task accuracy and trustworthiness.

A second-order point needs checking. Double backpropagation through ReLU networks returns zero curvature almost everywhere through the activations, although the penalty still shapes weights via the first-order path. We will measure whether a softplus backbone changes results, since this is exactly the non-smoothness limitation noted for automatic differentiation.

The analysis then asks what locality buys. It measures concept accuracy, and it measures leakage via the Oracle Impurity Score used by Concepts' Information Bottleneck Models. It measures intervention curves (task accuracy as increasing numbers of concepts are set to ground truth) and robustness on CUB + Places, where background shortcuts are the failure mode.

Baselines

The naive reference point is the joint concept bottleneck model of Concept Bottleneck Models. It is trained with concept and label losses only, on the same backbone, data split and epochs. That paper reports 19.9% task error on CUB against 17.5% for a black box, and Huang et al. report 77.2% accuracy for a ResNet18 CBM. Any version of our method must stay within about one point of this accuracy to count as a free improvement. We also run the independent model from the same paper, which avoids label leakage by construction but loses accuracy, to anchor the leakage axis.

The strongest competitor is ProtoCBM from On the Concept Trustworthiness in Concept Bottleneck Models. On ResNet18 it reports 78.6% accuracy, and across backbones it reports 62.8% to 70.5% on its Concept Trustworthiness Score. It is the strongest competitor because it targets exactly our quantity. We will rerun it from its published configuration on our split and give our method the same number of hyperparameter trials. We will report both our rerun and the paper's figures. Beating it means reaching at least its trustworthiness at equal or better accuracy on ResNet18 with a single added loss term. Matching it with a far simpler model is itself a useful result. We also include Concepts' Information Bottleneck Models as the strongest leakage-focused competitor. For CUB it reports gains in intervention monotonicity and OIS, but the file gives no scalar CUB numbers for it on Lacuna, so we rerun it at its chosen $\beta=0.5$ and a small sweep. We also include the Hard AR model of Addressing Leakage in Concept Bottleneck Models, which reports 81.4% concept accuracy.

Three ablations isolate the key ingredient. The first swaps the gradient penalty for a penalty on a Grad-CAM heatmap outside the part region; this keeps the supervision but drops exact sensitivity. The second trains with input masking, which also keeps the supervision but drops the gradient. The third shuffles the attribute-to-part map, which keeps the machinery but drops correct supervision. We also compare the random-projection estimator with the exact per-concept penalty at matched wall-clock time, and pixel-level placement with feature-map placement.

Why not just…?

Why not just mask everything outside the part region when training each concept head? This is the most serious threat. Masking is cheap and gives the detector no access to irrelevant pixels during training. It has two weaknesses. Test images arrive unmasked and without keypoints, so the detector sees a distribution it was never trained on. And masking constrains what the detector sees in training, not how it responds, so it may still react to background texture at test time. The masking ablation measures this directly under unmasked testing with occlusion-based locality. If masking matches our penalty, we will report it as the recommended method and drop the estimator claim.

Why not just use ProtoCBM? ProtoCBM already reaches 62.8% to 70.5% trustworthiness without spatial supervision, according to On the Concept Trustworthiness in Concept Bottleneck Models. Its cost is a specialized prototype architecture and four auxiliary losses, and the paper does not guarantee anything about leakage through the prototypes themselves. Our question is whether a single penalty on an ordinary bottleneck model gets there, and whether it lowers leakage and improves interventions, which ProtoCBM does not report. We also test the combination, because the two approaches are not exclusive.

Why not just penalize a cheap saliency map like Grad-CAM instead of the true Jacobian? Grad-CAM is a coarse, class-weighted summary of feature activations. The direction-2 literature documents that such post-hoc maps can reflect architecture artifacts rather than the model's actual dependence. A model can learn to keep its Grad-CAM tidy while remaining sensitive elsewhere. The Grad-CAM ablation, scored by occlusion rather than by Grad-CAM itself, will show whether the exact sensitivity matters.

Why not just reduce leakage globally with an information bottleneck? Concepts' Information Bottleneck Models compresses how much input information reaches concepts, but it is blind to where that information comes from, and it does not evaluate grounding. If the global penalty alone matches our trustworthiness and OIS, spatial locality adds nothing, and the head-to-head comparison is designed to reveal this.

Proposed Experiments

The primary benchmark is CUB-200-2011. We score task accuracy, concept accuracy, and the Concept Trustworthiness Score of Huang et al., computed from part bounding boxes. We also score a second locality score that the penalty never sees: we occlude the part region and, separately, the complement, and record the change in each concept logit. The robustness test uses CUB + Places, where Concept Bottleneck Models reports that CBMs hold at roughly 48–49% error while the standard model jumps to 62.7%. Background-insensitive concept detectors should push concept-model error further down. Leakage is measured by OIS and by intervention curves.

The decisive checks are these. First, occlusion locality must improve along with gradient locality. If only the trained-on gradient metric improves, the model has merely learned to hide its first-order sensitivity, and the claim fails. Second, the shuffled-part control must not help; if it does, the effect is generic regularization rather than grounding. Third, the whole-body attributes, which receive no penalty, should show no trustworthiness change, which confirms the effect is targeted. Fourth, at the $\lambda$ that matches the joint model's accuracy within one point, trustworthiness must approach ProtoCBM's reported range. Fifth, the estimator must reproduce the exact penalty's results at a fraction of the cost. We will report estimator variance, and the wall-clock overhead relative to standard joint training, alongside every accuracy number.