Exact-Order Concept Heads: Low-Rank Multilinear Predictors for Concept Bottlenecks

A concept bottleneck model first predicts a small set of human-readable attributes, such as "wing color is blue" or "beak is hooked", and then predicts the label from those attributes alone. Because the final step sees only the attributes, a person can read the model's reasoning and correct a wrong attribute at test time. The cost is the final step itself. In practice it is almost always a linear layer, which cannot express rules such as "hooked beak matters only when combined with large size and a dark crown." The alternatives are an opaque neural network on top of the concepts or an expensive symbolic search. We propose a third option for that final step: a multilinear polynomial over the concepts. Its interaction weights form a higher-order tensor, stored in a symmetric low-rank form so that all interaction orders up to a chosen maximum $D$ are computed in time linear in the number of concepts.

The main reason to use this form is that a multilinear function is affine in each concept separately. When a user flips one concept, the change in the prediction is therefore known exactly and in closed form. The prediction also splits exactly into a main-effect part, a pairwise part, a triple part, and so on. Binary concepts make this form fully expressive: any function of $k$ binary concepts is a multilinear polynomial of degree at most $k$. Truncating the degree at $D$ therefore gives a nested family that runs from the ordinary linear bottleneck ($D=1$) to an unrestricted predictor. This lets us answer a question the concept-bottleneck literature has not measured: how much of the gap between interpretable and black-box models comes from concept interactions, and at what order?

Core Claim

We claim three things. First, a concept head that is multilinear and of low rank, with order-wise centering to suppress spurious high-order credit, closes most of the accuracy gap between linear concept bottlenecks and neural concept heads on fine-grained benchmarks. Second, unlike a neural head, it gives an exact per-order decomposition of every prediction and exact intervention effects, with no extra inference cost. Third, on tasks whose true interaction order is known, the fitted head puts its predictive variance at that order rather than spreading it into higher orders.

Relation to Prior Work

The motivating evidence comes from the concept-bottleneck line. Concept Bottleneck Models introduced the two-stage concept-then-label design and test-time intervention. Its joint model reaches 19.9% error on CUB against 17.5% for a standard network. Auxiliary Losses for Learning Generalizable Concept-based Models reports standard CBM variants at 76–80% accuracy and closes the gap with auxiliary and orthogonality losses on the concept encoder. Both keep a linear head, and we change the head, not the encoder. Two further works recover accuracy by weakening the bottleneck itself. Concept Embedding Models replaces scalar concepts with high-dimensional embeddings, and Quantifying the Accuracy-Interpretability Trade-Off in Concept-Based Sidechannel Models shows that latent bypass channels invite the model to route around the concepts. Our proposal keeps the bottleneck strictly scalar and concept-only, and adds expressiveness only after it.

Two works add more expressive heads at a cost. Mixture of Concept Bottleneck Experts uses routed mixtures of linear or symbolic expressions, and its symbolic variant needs a combinatorial search that scales poorly with concept count. Energy-Based Concept Bottleneck Models models global concept–label dependencies through an energy function, at the price of optimization at test time. Our head is a single feed-forward contraction, and its interventions need no inner loop.

The method comes from the multilinear side. Scalable Higher-Order Tensor Product Spline Models shows that $D$-way interaction tensors can be factorized and evaluated recursively through Vieta-style elementary symmetric polynomials at cost linear in the number of features. It does this only for tabular regression, with no concepts or interventions, and we reuse that recursion. Interpretable Tensor Fusion identified higher-order interaction bias, the tendency of interaction terms to absorb signal that lower-order terms could carry, and controlled it with iterative batch normalization across three modalities. We transfer that concern to a setting with more than a hundred unordered concepts, where the risk is worse, and address it with centering and residual fitting by order. PolySAE adds quadratic and cubic interactions to sparse-autoencoder decoders. It is post hoc, unsupervised, and capped at order three, whereas our head is supervised, trained end to end, and intervenable. Sum-of-Parts proves that flat per-feature attribution can fail badly as dimension grows but keeps a flat sum over groups. We read that result as motivation for explicit interaction terms.

The tensor literature also constrains the design. Handling Slice Permutations Variability in Tensor Recovery shows that recovery with the t-product changes by more than 3 dB when slices are merely reordered. Reduced storage direct tensor ring decomposition shows that the cyclic ordering of modes in a tensor ring strongly affects storage and compute. Concepts have no natural order, so we deliberately avoid tensor-train and tensor-ring heads and use a symmetric CP form, which is invariant to how the concepts are ordered. Taken together, these papers make ordered tensor networks a poor fit for a set of unordered concepts.

The Model

Let $c \in [0,1]^k$ be the predicted concept probabilities and $\tilde c = c - \mu$ their deviations from the training-set means $\mu$. For class $y$ the head computes

$$ s_y(c) = b_y + \sum_{i=1}^{k} w_{y,i}\,\tilde c_i + \sum_{d=2}^{D}\sum_{f=1}^{F} a_{y,d,f}\; e_d\big(v_{1f}\tilde c_1,\dots,v_{kf}\tilde c_k\big), $$

where $e_d$ is the elementary symmetric polynomial of degree $d$, meaning the sum of all products of $d$ distinct arguments. The matrix $V=(v_{if}) \in \mathbb{R}^{k\times F}$ holds concept loadings on $F$ latent interaction factors and is shared across classes. The coefficients $a_{y,d,f}$ are class-specific weights for each order and factor, and $w$ and $b$ are the usual linear weights and biases.

Each $e_d$ contains no squared terms, so $s_y$ is multilinear. The Newton–Vieta recursion computes all $e_d$ up to order $D$ in $O(kFD)$ time, so no order-$D$ tensor is ever formed. The implicit weight tensor at order $d$ is symmetric with CP rank at most $F$. This is where the higher-order tensor direction enters, as a way to keep a $k^D$ object usable.

Multilinearity gives the exact intervention identity. Setting concept $i$ from $c_i$ to $c_i'$ changes every logit by $(c_i'-c_i)\,\partial s_y/\partial c_i$. That partial derivative does not depend on $c_i$ and can be split by order and by factor, so the head can say exactly why a correction changed the label.

Centering alone makes the order contributions orthogonal only when concepts are independent, which follows from the Hoeffding functional decomposition. CUB attributes are strongly correlated, so we add an ANOVA-style schedule in the spirit of the batch-normalization fix in Interpretable Tensor Fusion. Order $d$ is fitted to the residual left by orders below $d$, and then all orders are fine-tuned jointly with a penalty on the covariance between the contributions of different orders. We will measure how well this suppresses spurious high-order credit instead of assuming it does.

Research Program

The work has three parts. The first is building the head. We attach it to sequential and joint concept bottlenecks with a fixed ResNet-style backbone, use the concept encoder from Concept Bottleneck Models unchanged, and sweep $D \in \{1,\dots,6\}$ and $F \in \{4,16,64\}$.

The second is an order-sweep study. It reports, for each benchmark, the accuracy gain from each additional order and the share of logit variance each order explains, which yields an empirical profile of concept interaction order that no prior concept work has produced.

The third is an intervention study, which asks whether interventions through a multilinear head help as much as interventions through a neural head, and whether the closed-form order breakdown matches the counterfactual behavior of the neural head it is meant to replace.

Baselines

The naive reference point is the linear-head joint concept bottleneck from Concept Bottleneck Models, which is our model at $D=1$. On CUB that paper reports 19.9% error, against 17.5% for the standard network, and Auxiliary Losses for Learning Generalizable Concept-based Models reports standard CBMs at 76–80% accuracy. We rerun this baseline ourselves with our backbone so that the comparison is exact.

The strongest competitor on the specific question of head expressiveness is a concept bottleneck with a two-layer MLP head of matched parameter count. It sees the same concepts, can represent arbitrary interactions, and differs from our head only in transparency. We give it the same encoder, the same data, and the same hyperparameter search budget of 40 trials each.

The strongest published concept model on CUB accuracy in the file is coop-CBM with the concept orthogonal loss, at 84.1% accuracy against 82.3% for the black box (Auxiliary Losses for Learning Generalizable Concept-based Models). Its gain comes from the encoder side, so we also run our head on top of the coop-CBM encoder. That result shows whether the two gains add.

For interventions, the reference is Energy-Based Concept Bottleneck Models, which reports 71.3% overall concept accuracy under intervention on CUB against 39.6% for the concept embedding model. We compare task accuracy after intervention and wall-clock time per intervention. For the mixture-of-experts head of Mixture of Concept Bottleneck Experts, the file gives only relative figures, and we found no reported absolute CUB accuracy on Lacuna. We will run its linear variant under the same budget.

Four ablations isolate the key ingredients. The first is a pairwise-only factorization-machine head ($D=2$). The second is our head without centering or residual scheduling. The third is a tensor-train head of equal parameter count, trained under five random concept orderings, to test the permutation sensitivity predicted by the slice-permutation and tensor-ring results. The fourth is a full-rank head at $D=2$, to separate the effect of the low-rank constraint from the effect of the interactions themselves.

We count the proposal as beating the baselines if three things hold on CUB. It must close at least half the accuracy gap between the linear-head and MLP-head bottlenecks. Its accuracy after intervention must be within one point of the MLP head's. And it must do both while giving exact order decompositions and costing less per intervention than ECBM.

Why not just…?

Why not just use an MLP head on the concepts? This is the real threat on accuracy, and we expect the MLP head to match or slightly exceed ours. The question is what is lost. An MLP head's response to an intervention depends on the values of the other concepts in ways nobody can inspect, and it has no exact split by order. We test whether this matters by comparing post-hoc interaction attributions of the MLP head with its actual responses to counterfactual interventions. If those attributions predict intervention effects as well as our closed form does, and the MLP head is also more accurate, our advantage shrinks to convenience, and we will report it that way.

Why not just fit a pairwise factorization machine? If the order sweep shows that $D=2$ captures nearly all of the gain on CUB, CelebA, and AwA2, then the higher-order claim fails on natural data and the contribution becomes the finding that concept interactions are essentially pairwise. The synthetic XOR, trigonometric, and dot-product tasks from Concept Embedding Models, together with parity-style tasks of order 3 to 5 that we construct over concepts, show whether higher orders are recoverable when they exist. The natural benchmarks show whether they exist at all.

Why not just keep the linear head and improve the concept encoder, as coop-CBM does? Encoder improvements raise concept accuracy, from 93.9% to 97.3% in Auxiliary Losses for Learning Generalizable Concept-based Models, but a linear head still cannot express a label that depends on a conjunction of concepts, however accurate the concepts are. The experiment with our head on the coop-CBM encoder tests whether the two gains add. If they do not, the gap was encoder error rather than head expressiveness.

Why not just use concept embeddings or a side channel for accuracy? Both recover accuracy by letting information bypass the named concepts. That is exactly the failure that Quantifying the Accuracy-Interpretability Trade-Off in Concept-Based Sidechannel Models measures with its independence score. Our head keeps every path from input to label running through scalar concepts. We report accuracy under full intervention as the fair common ground between these approaches.

Proposed Experiments

We use CUB-200-2011 with its 112 concepts, CUB+Places for background shift (the file reports CBM error of roughly 48–49% against 62.7% for the standard network, from Concept Bottleneck Models), CelebA, AwA2, and the synthetic tasks above. The metrics are task accuracy, the share of logit variance explained by each order, accuracy curves as concepts are corrected in random order and in order of uncertainty, wall-clock time per intervention, and recovery of the true order on the synthetic tasks.

Three checks decide the question. The first is whether, on synthetic tasks with known order $D^*$, the centered and residually fitted head puts most of its variance at order $D^*$ while the uncentered ablation leaks it upward. That check tests the control of interaction bias directly. The second is whether the order sweep on CUB shows real gains beyond $D=2$. The third is whether, under background shift, the multilinear head keeps the robustness advantage of concept models over black boxes or loses it by fitting brittle high-order conjunctions. A positive result on all three would show that much of the accuracy the concept-bottleneck literature has traded away can be recovered with interactions we can read and correct exactly.