Concept Bottlenecks With Identifiable Interaction Heads

A concept bottleneck model predicts a label in two steps. First a neural network reads the input (an image, a patient's vital signs) and outputs a short list of human-readable concepts such as "wing is red" or "mild hypotension". Then a small, transparent model turns those concepts into the final prediction. The design is meant to let an expert correct a wrong concept at test time and watch the prediction update. In nearly every such model, that second step is a linear scorer that adds up concept contributions independently. It therefore assumes concepts never interact, and the multi-criteria decision literature shows this assumption breaks routinely. Under the linear scorer, a formal blazer plus casual shorts scores well, and a large lead on one criterion compensates in ways a simple sum cannot represent. Making the scorer interaction-aware looks like an obvious fix. It carries a hidden cost. Real concepts are strongly correlated, so many concept combinations never appear in training. An interaction model's values on those empty combinations are unconstrained by data. They are also exactly where a human intervention can send the input.

We propose a concept-to-label head built from a few small interaction terms, each over one, two, or at most three concepts. Every term is centered against the observed concept distribution, so the decomposition into terms is unique wherever data exist. Where data do not exist, the head falls back to its lower-order terms and says so. The claim is that this head recovers the accuracy lost by linear heads on hard (binary) concept bottlenecks. It should also produce explanations that are reproducible across random seeds and data resamples, which neither neural additive models nor unconstrained interaction heads achieve. Finally, it should turn test-time interventions from a silent extrapolation into a check whose support is known. We evaluate all methods under a matched-tuning, stratified-split protocol, adopting the audit discipline that overturned published rankings in clinical coding.

Core Claim

Three findings in the file motivate the work. Concept Bottleneck Models introduced the two-stage design with a linear concept-to-label map. It reports a joint model error of 19.9% against 17.5% for an unconstrained network on CUB, so a small but real accuracy gap remains. Addressing Leakage in Concept Bottleneck Models showed why "soft" bottlenecks look accurate. Continuous concept probabilities smuggle label information to the scorer, and interventions then hurt. Their fix is a hard autoregressive concept predictor plus a separate latent side channel. It raises CUB concept accuracy to 81.4% from 48.1% for the soft independent model and stays within about 1% of soft label accuracy. That fix models concept dependence only on the input side, inside the predictor from input to concepts, and it uses an arbitrary fixed concept ordering. The label head stays additive, and the remaining accuracy is carried by a side channel that no one can inspect.

The third finding constrains the design. Challenges in interpretability of additive models shows that under correlated inputs ("concurvity", the nonlinear analogue of collinearity), additive models admit many different shape functions with the same predictions. Orthogonality and concurvity penalties do not remove this ambiguity. Two researchers training the same model can therefore publish contradictory explanations.

The method source is Learning GAI-Decomposable Utility Models for Multiattribute Decision Making. Its SMKGAI method learns sparse interaction cliques using kernels whose components integrate to zero, following a functional ANOVA (analysis-of-variance) decomposition. That makes the decomposition unique on small tabular problems with independent-looking inputs. Its zero-integral constraint is taken with respect to a product measure, however. With correlated binary concepts, the product measure places weight on cells the data never visit, so uniqueness on paper does not become stability in practice.

Our specific contribution is to center the interaction terms against the empirical concept distribution rather than a product measure. We also treat unsupported cells as an explicit, reportable part of the model, not something to hide.

Research Program

The head takes hard concepts $c \in \{0,1\}^k$ and outputs a logit.

$$f(c) = b + \sum_{S \in \mathcal{S}} f_S(c_S),$$

Here $\mathcal{S}$ is a learned sparse collection of concept subsets of size at most three, and $f_S$ is a lookup table over the $2^{|S|}$ configurations of the concepts in $S$. Each term must satisfy hierarchical centering under the training concept distribution $p$.

$$\mathbb{E}_{p}\left[f_S(c_S)\,\middle|\,c_{S\setminus\{j\}}\right] = 0 \quad \text{for every } j \in S,$$

This means no interaction term can absorb an effect that a lower-order term could carry. The condition removes the constant-shifting ambiguity. Because it is weighted by $p$, it also pins down each term on every cell that has data. We implement it by a projection step after each gradient update, a weighted backfitting-style purification. We then verify it numerically rather than trusting it.

Structure is selected with a group-sparsity penalty over candidate subsets. The penalty is strong-hierarchy-gated: a pair is eligible only if both singletons are active. We screen pair and triple candidates by the mutual information of their concepts with the label residual, which keeps the search tractable at $k=312$. For any cell with fewer than $m$ training examples, the head sets that cell's interaction value to zero, so the prediction there reduces to lower-order terms, and it marks the prediction as low-support.

The head sits on top of the Hard AR concept predictor, reused as-is from Addressing Leakage in Concept Bottleneck Models. The first question we ask is whether an inspectable interaction head can replace some of what their side channel currently carries. Measured on their information-theoretic completeness score, this is the proposal's accuracy claim.

The reproducibility side of the program is an explanation-stability protocol. Every method is trained over ten seeds and five stratified resamples. We then measure agreement of the learned explanations: rank correlation of term importances, sign agreement of each concept's main effect, and Jaccard overlap of selected interaction subsets. This follows the lesson of Automated Medical Coding on MIMIC-III and MIMIC-IV. That audit showed uneven threshold tuning let an older Bi-GRU beat newer architectures, and unstratified splits dropped rare labels from test sets entirely. Every method in our study therefore gets the same tuning budget, tuned decision thresholds, and stratified splits.

Baselines

The naive reference point is the hard concept bottleneck with a linear logistic head, as in Concept Bottleneck Models and the user study of Are Concept Bottleneck Models Effective as Decision-Support Systems?. It is the default practitioners deploy, and it embodies the independence assumption we are testing. Beating it means higher label accuracy at equal concept accuracy. On CUB, the target is to close a meaningful share of the 2.4-point gap between the reported 19.9% joint error and the 17.5% black-box error.

The strongest existing method is Hard AR with its side channel, from Addressing Leakage in Concept Bottleneck Models. Its reported numbers are 81.4% concept accuracy on CUB and label accuracy within about 1% of leaky soft models. On MIMIC-III EWS, no reported numerical result was found on Lacuna, so we will reproduce it ourselves before comparing. We run Hard AR with and without its side channel, using the same backbone, data, and number of tuning trials as our head. Beating it has three parts: matching its label accuracy while using a smaller side channel, or none; higher explanation stability across seeds; and intervention curves that stay monotone. Hard AR's label-accuracy advantage, if any, is our main bar.

The unconstrained interaction head is an MLP or gradient-boosted trees on the hard concepts. It bounds how much accuracy concept interactions can buy at all.

A pairwise boosted additive head with post-hoc purification under a product measure represents the closest practical interpretable competitor. It is described here from standard practice; the file contains no specific paper for it.

Two ablations isolate the key ingredient. The first swaps our distribution-weighted centering for product-measure centering in the style of SMKGAI. The second removes the low-support fallback. If neither ablation lowers stability or intervention reliability, the proposal's core claim fails.

Why not just…?

Why not just put an MLP on the hard concepts? It will probably match or exceed our accuracy. That is precisely why it is a baseline and serves as the accuracy ceiling. It gives up the property that motivated a bottleneck in the first place: a person can read why concepts combine into the label and check what an intervention will do. If our head gets within noise of the MLP while keeping a readable, stable decomposition, the MLP loses on the question asked here. If the MLP beats us by a wide margin, that is evidence that concept interactions of order three or less are not where the missing accuracy lives, and we will report it.

Why not just use a pairwise boosted additive model on the concepts? This is the real threat, since it already models pairwise interactions and can be purified. Its purification is typically defined against a product or marginal measure. Our argument, following Challenges in interpretability of additive models, is that correlation makes such decompositions unstable. The product-centering ablation and this baseline test that argument directly. If the boosted head is equally stable across seeds and equally reliable under intervention, our contribution shrinks to the support-flagging mechanism, and we would say so.

Why not just average explanations over many seeds? Averaging hides instability rather than removing it. The averaged explanation is not the explanation of any deployed model, and interventions act on one deployed model. We include a ten-seed-averaged linear head in the stability analysis, measuring how far each single model's terms sit from that average.

Why not just keep Hard AR's side channel and stop? The side channel restores accuracy by carrying information no human can inspect or intervene on. A larger side channel means less of the decision passes through concepts. The measurable question is how much side-channel capacity the interaction head lets us remove at fixed accuracy, tracked by their completeness score.

Proposed Experiments

The first experiment is a synthetic sanity check with known answers. We use the spline and high-interaction functions from SMKGAI, for which it reports roughly a tenfold MAE reduction over additive models. We pass their inputs through a correlated binarization so that some concept cells are empty, which lets us measure interaction recovery (false discovery rate) and term stability as correlation grows.

The main experiments use CUB with 312 attributes and MIMIC-III EWS, the vital-signs early-warning cohort from Addressing Leakage in Concept Bottleneck Models, rebuilt with stratified splits. We report label accuracy, concept accuracy, completeness, the three stability measures above, and intervention curves: task error as increasing numbers of concepts are corrected to ground truth. On OAI, correcting two concepts cut error from over 0.4 to about 0.3 in Concept Bottleneck Models, and we include OAI if access permits. CUB + Places tests whether interaction terms reintroduce background shortcuts. There the reported black-box error is 62.7% against roughly 48–49% for bottleneck models.

The decisive check targets interventions directly. For every intervention, we record whether the corrected concept vector lands in a cell with fewer than $m$ training examples for any active term, and we compare error on flagged versus unflagged interventions across all heads. The proposal is supported if three things hold together. Unconstrained interaction heads show sharply worse error on low-support interventions. Our head's flag predicts where errors concentrate. Our head's term importances agree across seeds substantially better than both the product-centered ablation and the boosted pairwise head. If stability does not improve, or low-support interventions turn out harmless, then correlated concepts are not the practical obstacle we claim, and the linear head plus side channel remains the better choice.