Separating Hidden Context from Local Rules in Concept Bottleneck Models

Concept bottleneck models make a prediction in two steps. First they predict human-named attributes of an input, such as "yellow belly" or "joint space narrowing", and then they predict the label from those attributes alone. A human can read the attributes, and can also correct a wrong one at test time. Real attributes are correlated, so recent work learns a graph of dependencies among concepts. That graph lets one corrected concept update the others.

We argue that these graphs mix two different things. One is broad hidden context: a bird photographed on water tends to show several unrelated "aquatic" attributes at once. The other is a small number of genuine local rules, such as one beak attribute implying another. When a single sparse graph has to absorb both, it either fills with spurious edges or drops real ones.

We propose to borrow the classic fix from statistics, where a dependency structure is split into a sparse part plus a low-rank part. The sparse part becomes a short, readable list of concept-to-concept rules. The low-rank part becomes a handful of hidden context factors that sit inside the concept layer and never reach the label directly. The bet is concrete: at equal task accuracy and equal intervention gains, this split should yield a far sparser and more stable concept graph than current graph-based bottlenecks, and its hidden factors should line up with known nuisance variables such as image background.

Core Claim

Latent variable graphical model selection via convex optimization supplies the method. It shows that when some variables are unobserved, the precision matrix of the observed variables (the inverse covariance, whose zero pattern encodes conditional independence) becomes dense. The paper shows this dense matrix can be recovered as a sparse matrix minus a low-rank one, $K = S - L$. Here $S$ holds direct conditional dependencies, and $L$, which is positive semidefinite, captures the diffuse influence of the hidden variables. Recovery is exact when two conditions hold: the sparse graph has bounded degree, and the hidden factors spread their influence over many variables (incoherence).

Our claim is that concept spaces fit this setting closely. The concept annotations are observed, while the unannotated scene and context variables are hidden. Placing the decomposition inside the bottleneck should therefore move the trade-off curve between graph sparsity and accuracy, not just move a point along it.

The proposal is distinct from the closest prior work. Graph Concept Bottleneck Models learns one concept graph with contrastive learning, a sparsity penalty and graph-network message passing, and reports CUB accuracy of 77.95% against 74.69% for its non-graph baseline. It has no low-rank term, so whatever correlation hidden context induces must be expressed as edges. Relational Concept Bottleneck Models passes messages over hand-templated logical relations between entities, again without any global latent component.

Recommender work such as Ordinal Random Fields for Recommender Systems and Preference Relation-based Markov Random Fields for Recommender Systems motivates the global-plus-local split. It shows that adding local pairwise potentials to a low-rank model helps (MAE 0.6870 versus 0.6918 on MovieLens-1M). However, its global factors are uninterpretable, and it has no concept layer a human can intervene on. We move that global-plus-local structure into a space where the local part is made of named concepts.

Mechanism

A concept predictor $g$, built on a frozen or fine-tuned image backbone, outputs concept logits $\hat c$ for $k$ concepts. A refinement layer treats these logits as noisy observations from a concept distribution with mean $\mu$ and precision $K = S - L$. When a human fixes a subset $O$ of concepts to values $c_O$, the remaining concepts $U$ are updated by Gaussian conditioning:

$$\tilde c_U = \mu_U - K_{UU}^{-1} K_{UO}\,(c_O - \mu_O),$$

where $K_{UU}$ and $K_{UO}$ are blocks of $K$. Without any intervention, the same layer shrinks each instance's noisy logits toward a consistent configuration.

The update has a natural explanation. Each change to an unintervened concept splits exactly into a part carried by named edges in $S$ and a part carried through the rank-$r$ factors of $L$. The model can therefore say which correction came from a local rule and which came from "context factor 2 shifted".

A linear predictor $f$ maps the refined concepts to the label, as in Concept Bottleneck Models. Training is joint:

$$\min_{g,f,\mu,S,L}\; \mathcal{L}_{\text{task}} + \alpha\,\mathcal{L}_{\text{concept}} - \beta\,\ell(S-L;\hat c) + \lambda\big(\gamma\|S\|_1 + \mathrm{tr}(L)\big).$$

The terms are as follows. $\ell$ is the Gaussian log-likelihood of the concept logits. $\|S\|_1$ is the elementwise absolute sum that makes $S$ sparse. $\mathrm{tr}(L)$ is the trace of $L$, which pushes it toward low rank. $\gamma$ sets how much correlation goes into edges versus factors, and $\alpha$, $\beta$, $\lambda$ are loss weights.

The design keeps $L$ from becoming a side channel that bypasses the bottleneck. $L$ only changes concept values, and every refined concept is still scored against its annotation through $\mathcal{L}_{\text{concept}}$. We therefore report concept accuracy alongside task accuracy throughout.

Binary concepts are not Gaussian. Our first-line answer is to work in logit space. As a check, we also fit an Ising version with pseudo-likelihood, the same workaround the ordinal random field paper uses for its pairwise potentials.

Research Program

The work proceeds in three stages.

The first stage tests identifiability on synthetic concept data. We plant a sparse graph and a few hidden factors with controlled degree and incoherence, then sample images-as-feature-vectors whose concepts follow the model. This tells us when edge recovery succeeds or fails before any real data is involved. Success means recovering the planted edges and the number of factors.

The second stage fits the decomposition offline to concept annotations on CUB-200-2011 (112 attributes) and AwA2. We ask whether the sparse part is markedly sparser than a graphical-lasso fit at equal held-out likelihood. We also inspect whether the edges are semantically sensible.

The third stage is end-to-end training with interventions on CUB, AwA2, CUB+Places, and OAI (where its concept annotations can be obtained). CUB+Places is the decisive real setting because its hidden variable is known: background scene was pasted in by construction. We test whether the column space of $L$ aligns with background labels, which are withheld during training. We then test whether zeroing $L$'s contribution at test time reduces the background-shift error.

Baselines

The naive reference point is the joint concept bottleneck of Concept Bottleneck Models, which treats concepts as independent. That paper reports a joint-bottleneck error of 19.9% on CUB against 17.5% for an unconstrained model, and roughly 48–49% error versus 62.7% on CUB+Places. We rerun both models on our backbone so that the comparison isolates the refinement layer. The unconstrained black box on the same backbone serves as the accuracy ceiling.

The strongest existing competitor is Graph Concept Bottleneck Models, with a reported 77.95% CUB accuracy for its label-free variant. It is the only method in the file that learns concept dependencies and uses them for intervention, which is exactly the capability we claim to improve. Its reported number comes from a label-free CLIP-concept pipeline. We will therefore reimplement it on our backbone and our annotated concepts, with the same training data, the same hyperparameter search budget, and a sweep of its sparsity weight. This produces a full accuracy-versus-edge-count curve rather than a single point. Beating it means one of two things. Either our curve reaches its best accuracy and intervention gain with substantially fewer edges, or it gives higher post-intervention accuracy at matched edge count. Edge sets must also be more stable under bootstrap resampling.

Two ablations isolate the key ingredient. In the sparse-only version we set $L=0$, which is essentially a graphical-lasso concept layer. In the low-rank-only version we set $S$ to be diagonal, which is a factor-model shrinkage layer. A third reference is a dense, unregularized precision matrix, which is the most accurate but least readable option. Every variant shares the backbone, predictor, and tuning budget.

Why not just…?

Why not just tune the sparsity penalty of an existing concept graph? This is the real threat. If CUB's concept correlations are not driven by diffuse hidden context, the sparse-only ablation will trace the same frontier as the full model. The low-rank part would then add nothing. Chandrasekaran et al. show that graphical lasso returns dense, spurious graphs when hidden variables exist. Whether concept spaces actually have such variables is an empirical question, and the frontier comparison settles it directly. We expect the gap to appear most clearly on CUB+Places, where a hidden background variable is planted by construction. If no gap appears there either, the hypothesis is wrong.

Why not just add the missing context as more named concepts? Hybrid Concept Bottleneck Models takes this route. It learns extra concept vectors and translates them into text with GPT-2, reporting 68.38% on CIFAR-100 against 70.1% for a linear probe. That approach makes hidden context explicit, but it depends on a translator that can hallucinate. It also gives no guarantee that added concepts remove spurious edges. We include an oracle version: CUB+Places with the true background added as a concept. If the low-rank model approaches this oracle without the label, that is evidence that $L$ recovers named context for free. If it falls well short, explicit concepts are the better route.

Why not just use a dense graph or concept embeddings and accept lower readability? Deep concept reasoning: beyond the accuracy-interpretability trade-off shows that richer concept representations remove accuracy penalties. If accuracy were the only goal, the dense-precision baseline would suffice. Our claim concerns the readability of the dependency explanation at fixed accuracy. The dense baseline anchors the accuracy end of the frontier, and we report how much accuracy it buys per extra edge.

Proposed Experiments

On synthetic data, we sweep graph degree and factor incoherence. We measure edge-recovery F1 and rank recovery for the full model against the sparse-only ablation, and expect the full model to win only when hidden factors are diffuse, as the theory predicts.

On CUB and AwA2, we report three things. The first is task accuracy, concept accuracy, and edge count along each method's regularization path. The second is intervention curves: task error and error on unintervened concepts as we correct 1 to 20 concepts, in random order and in uncertainty order. The third is edge stability across bootstrap fits.

On CUB+Places, the decisive checks are threefold. The first is the alignment between $L$'s factors and the withheld background labels. The second is the change in shifted-test error when $L$ is removed. The third is whether the sparse graph loses the water-related spurious edges that the sparse-only ablation keeps.

The proposal fails if the full model's curve is indistinguishable from the sparse-only ablation on both CUB and CUB+Places. It is confirmed if it matches the reimplemented graph bottleneck's accuracy and intervention gains with a clearly smaller and more stable edge set, and its hidden factors recover the background.