Concept Leakage as a Side Effect of Margin Maximization

Concept bottleneck models are image classifiers built to be readable. The network first predicts a set of human-named attributes, such as "red wing" or "bone spur," and then predicts the label from those attributes alone. Their known weakness is leakage: when the attribute predictions are soft probabilities, the network hides class identity in their small fluctuations. For example, it might output 0.91 rather than 0.92 for an unrelated attribute depending on the species. The label predictor learns to read these hidden codes, so the attributes stop meaning what they say, and correcting them by hand at test time can make predictions worse. This proposal argues that leakage is not an arbitrary quirk. It follows from a well-understood property of training with cross-entropy: gradient descent keeps pushing the classifier's confidence margin upward forever, even after every training example is classified correctly. The continuous concept layer is the cheapest place to buy that extra margin.

If the claim holds, it makes three testable predictions. First, leakage should grow during the late phase of training, after training error reaches zero. Second, it should track the task head's normalized margin. Third, it should largely disappear when the task loss stops rewarding margin beyond a threshold. That fix costs one loss substitution. The current remedy, from Addressing Leakage in Concept Bottleneck Models, instead hardens the concepts and adds a side channel plus an autoregressive concept predictor.

Core Claim

In a jointly trained bottleneck model, an encoder $g$ maps an image $x$ to concept probabilities $\hat c = \sigma(g(x)) \in [0,1]^k$, and a linear head $f(\hat c) = W\hat c$ predicts the label $y$. The model is trained on

$$\mathcal{L} = \mathrm{CE}(y, W\hat c) + \lambda\, \mathrm{BCE}(c, \hat c),$$

where $c$ are the true binary concepts and $\lambda$ weights concept supervision. This is the joint protocol of Concept Bottleneck Models. Cross-entropy has an exponential tail, so its gradient never quite vanishes on separable data. Gradient Descent Maximizes the Margin of Homogeneous Neural Networks shows that the parameters then drift toward a maximum-margin direction, and Cai et al. extend this to networks with biases and residual connections.

Two facts combine here. The concept loss is satisfied once each $\hat c_j$ lands on the right side of 0.5, and it is nearly flat near saturation. The task loss, by contrast, still rewards any perturbation of $\hat c$ that is aligned with the class.

Puli et al. proved the analogous result for ordinary classifiers. Max-margin training prefers to combine easy "shortcut" features with noise over relying on the stable feature, because the combination reaches a given margin at smaller norm. We claim that within-concept, class-correlated deviations play exactly the shortcut role inside the bottleneck. They are cheap margin when the true concepts are not a sufficient statistic for the label, which Margeloiu et al. call the Markovian gap. The neural-collapse results support the same picture from another angle. Ji et al. and Min et al. show that max-margin training squeezes the representation feeding a linear head toward class means. In a bottleneck, that means concepts become class signatures rather than sample-level attributes.

The proposed remedy is to bound the margin of the task head only, leaving concept supervision untouched. We replace $\mathrm{CE}$ with a margin-control loss from Puli et al., such as σ-stitch (which penalizes margins above a threshold $\tau$) or spectral decoupling (which penalizes $\|W\hat c\|_2^2$), and we apply it only on the $c \to y$ path.

Research Program

The theory component studies a stylized model. Features are fixed, the concept head is linear-logistic, and the true concepts explain the labels only partially, with a tunable residual. In this model we aim to prove two things. Under gradient flow on the joint loss, the component of concept logits that is orthogonal to the true concepts but aligned with the label grows without bound in normalized terms. Under a margin-capped task loss, that component stays bounded by a function of $\tau$ and the Markovian gap. We treat the growth rate as an open question. Our working hypothesis is logarithmic growth in $t$, by analogy with the $O((\log t)^{1/L})$ norm growth Lyu and Li derive. The model's dependence on the optimizer, through its margin geometry, gives a secondary prediction. Adam's $\ell_\infty$ bias (Zhang et al.) should spread leakage across many concepts. Plain gradient descent's $\ell_2$ bias should concentrate it.

The empirical component trains joint models on CUB-200-2011 with its 112 attributes, on CUB with Places backgrounds, and on the synthetic task. Each model is trained well past zero training error, with checkpoints along the way. At each checkpoint we track four quantities:

  • the task head's normalized margin;
  • a concept-space collapse statistic, defined as within-class concept variance divided by between-class variance;
  • leakage, defined as how accurately a fresh linear probe recovers the label from $\hat c$ restricted to samples sharing the same true concept vector pattern, measured on held-out data relative to chance;
  • intervention gain, defined as accuracy after replacing $m$ predicted concepts with ground truth, for $m$ from 0 up to the full set.

Starter code for the bottleneck variants is available in mvandenhi/scbm and ejkim47/prob-cbm.

Baselines

The naive reference point is the standard joint model trained with cross-entropy for a fixed long schedule. Koh et al. report its task error on CUB as 19.9%, against 17.5% for a black-box model, and we reproduce both. The sequential and independent protocols from the same paper serve as no-leakage-path references. They show what accuracy is lost when the task gradient cannot reach the encoder.

The strongest existing method is the hard autoregressive model with a continuous side channel from Margeloiu et al.. On CUB they report 81.4% concept accuracy, versus 48.1% for the soft independent model, with label accuracy within about 1% of leaky soft models. Every method we compare is run with the same backbone, data splits, epoch budget and number of hyperparameter trials, using their released design where possible. This includes our $\tau$ and $\lambda$ sweeps and their side-channel width. A margin-capped joint model beats it if it matches its concept accuracy and intervention curve within noise while either reaching higher task accuracy or matching it without hard concepts, side channel, or autoregressive decoder.

Two ablations isolate the key ingredient. In the first, the margin cap is applied to the concept loss instead of the task head; we predict this does not reduce leakage. In the second, the cap is applied to both. A further control uses the same training budget with constant versus decaying learning rate. This separates "time spent in the terminal phase" from "number of epochs."

Why not just…?

Why not just stop training early? This is the most serious threat. If leakage appears only after zero training error, early stopping on validation task accuracy might remove most of it at no cost. The checkpoint trajectories settle this directly. If leakage is already large at the early-stopping point, early stopping is insufficient and our mechanism is at best partial; we would report that. If leakage is small there but task accuracy keeps improving afterward, margin capping is the only option that keeps the later gains without the leakage. Either outcome is informative about the mechanism.

Why not just use hard binary concepts? Hard concepts close the leakage channel by construction. However, Margeloiu et al. found that hard models lose task accuracy whenever the concept set is incomplete, and needed a side channel and an autoregressive predictor to recover. We include their full system as the strongest baseline rather than arguing past it.

Why not just use weight decay or label smoothing? Puli et al. argue that explicit $\ell_2$ penalties still favor shortcut-plus-noise solutions, since those have smaller norm, so we expect weight decay alone to fail. Label smoothing is different. It is itself a margin-bounding loss, and if it works as well as σ-stitch, that confirms the hypothesis rather than refuting it. We run both, at matched tuning budget, and report label smoothing as a member of the margin-control family.

Proposed Experiments

The decisive check is a correlation test across all runs and checkpoints on CUB. Leakage should rise with the task head's normalized margin, and intervention gain should fall with it. Margin capping should break both trends while concept accuracy is unchanged. The causal test is on the synthetic task. Varying the Markovian gap should change leakage under cross-entropy but not under the capped loss, matching the theory's prediction.

On CUB with Places backgrounds, Koh et al. report roughly 48–49% error for the standard bottleneck model versus 62.7% for the black box. We test whether reducing leakage improves robustness to background shift further, which it should if leaked codes partly encode background. For the optimizer-geometry prediction we compare SGD and Adam at matched accuracy. Under Adam, leakage should be spread across more concepts. For worst-group-style evaluation we have only Puli et al.'s roughly 10% worst-group gain on Waterbirds for non-bottleneck models; no reported result was found on Lacuna for bottleneck models on Waterbirds.