Firm-Thresholded Residuals for Leak-Resistant Hybrid Concept Bottlenecks

A concept bottleneck model is a classifier that first predicts human-readable attributes ("red wing", "hooked beak") and then predicts the label from those attributes alone. This design lets a person fix a wrong attribute at test time and watch the prediction change. The cost is accuracy, because no hand-written attribute list covers everything that matters. The usual repair adds a free "residual" channel of unlabeled features next to the concepts. The downstream classifier then learns to route information through that channel, and correcting a concept stops changing the answer.

We propose to control this leak with the shape of the function applied to the residual channel. We borrow the firm threshold from signal denoising. It zeroes small values, ramps continuously through a transition band, and passes large values through unchanged. Our claim is that this shape lets the residual contribute only when it carries a large, confident signal, and that it does so without the magnitude bias of $\ell_1$ penalties or the gradient problems of hard gates. If the claim holds, hybrid concept models get a better trade-off between accuracy and intervenability than either existing sparsity choice allows.

Core Claim

In wavelet denoising, soft thresholding subtracts a constant from every surviving coefficient. This blurs peaks (Ting et al.). Hard thresholding keeps magnitudes but jumps at the threshold (Zhang et al., Lin et al.). Hybrid operators that tend to the identity line for large inputs beat both (Wang et al., Chen et al.). Firm thresholding is the closed-form proximal step of the minimax concave penalty (Li et al.). That paper shows it removes the $\ell_1$ bias in sparse system identification, gaining 2 to 5 dB in normalized mean squared error.

We argue the same bias drives leakage in hybrid concept models. Suppose the residual is penalized with $\ell_1$. To recover accuracy, training must lower the threshold, because surviving residual values are shrunk and underweighted. A lower threshold admits many small residual activations. Those small activations are exactly the cheap side channel the classifier exploits.

An unbiased operator should reach the same accuracy at a higher threshold. At that threshold, fewer examples use the residual at all, so concept interventions keep their force. The testable prediction is specific. At matched task accuracy, a firm-thresholded residual should show lower residual support and higher intervention gain than an $\ell_1$ residual, and should train more stably than a hard-gated residual.

Mechanism

Let $c(x)\in\mathbb{R}^k$ be the $k$ concept scores, $r(x)\in\mathbb{R}^m$ the $m$ residual features, and $\hat y = W_c\,c(x) + W_r\,\psi(r(x);\lambda,\alpha)$ a linear head. Here $\psi$ acts elementwise with lower threshold $\lambda$ and upper threshold $\alpha>\lambda$:

$$\psi(r)=\begin{cases}0 & |r|\le\lambda\\ \mathrm{sign}(r)\,\dfrac{\alpha(|r|-\lambda)}{\alpha-\lambda} & \lambda<|r|\le\alpha\\ r & |r|>\alpha\end{cases}$$

We also test a variant with continuous first derivative, following the observation that kinks in the derivative cause residual artifacts (Jiao et al.).

Both thresholds receive ordinary gradients from examples that fall in the transition band. Learning them therefore needs no surrogate gradient. This is the operational difference from JumpReLU, whose threshold is learnable only through a kernel-density straight-through estimator with a sensitive bandwidth (Jumping Ahead).

Training minimizes task loss plus concept loss, as in the joint scheme of Concept Bottleneck Models. We add a smooth penalty on the expected number of residual coordinates in the transition band or above it.

A secondary variant makes $\lambda$ depend on the input. When the concept predictor is confident, $\lambda$ rises, so the residual speaks mainly when concepts are uncertain. This mirrors how Cao and Li adjust threshold shape per wavelet scale from the noise-to-signal energy ratio.

Relation to Prior Work

Three works motivate the gap. Concept Embedding Models and Deep concept reasoning recover accuracy with rich concept embeddings and random intervention training. They do not shape an explicit residual channel; we keep scalar concepts and move the capacity question to the residual. Hybrid Concept Bottleneck Models add a learnable dynamic concept bank regularized by orthogonality and Sinkhorn alignment, but apply no sparsity operator to it. That is the closest architecture, and we insert our operator exactly where it lacks one.

Three works constrain the design by showing what the obvious sparsity choices cost. Sparse-CBM finds that $\ell_1$ on bottleneck weights shrinks true activations and trails a Gumbel-softmax relaxation, which in turn has high gradient variance. CB-SAE prunes low-utility dictionary neurons with a hard post-hoc cutoff. JumpReLU removes shrinkage bias in sparse autoencoders but needs straight-through gradients and produces dense features.

The obvious intersection idea would be "a smooth firm threshold as a sparse-autoencoder activation." We deliberately do not pursue it. It would be a small variation on JumpReLU and is threatened by simple debiasing tricks, and the file reports no gap it would close that JumpReLU leaves open in fidelity. The leakage question in concept models is where the magnitude-shape argument makes a distinct, falsifiable prediction.

Research Program

We build hybrid concept models in two regimes.

The first uses annotated concepts. It follows Koh et al. on CUB-200-2011 (112 attribute labels), CUB+Places, and CelebA, with a ResNet backbone. Here ground-truth interventions are possible.

The second uses CLIP concept banks, following HybridCBM on CUB and CIFAR-100. Here the residual is the dynamic bank.

In both regimes, only the residual operator changes. The options are none, $\ell_1$/soft, hard gate with straight-through estimator, firm, $C^1$ firm, and input-adaptive firm.

For each operator we sweep its sparsity strength and trace a frontier of task accuracy against interpretability. We measure interpretability in three ways:

  • Intervention gain: accuracy improvement after correcting the $j$ most uncertain concepts, for $j$ up to 10.
  • Residual support: mean number of active residual coordinates, and the fraction of test examples with any active residual.
  • Concept sufficiency: accuracy with the residual zeroed.

We also probe duplication. A linear probe tries to predict concept labels from $\psi(r)$; high probe accuracy means the residual is re-encoding concepts, which is leakage.

The analytic component tests the mechanism directly. For $\ell_1$ and firm models at matched accuracy, we compare the histogram of surviving residual magnitudes and the learned $\lambda$. The hypothesis predicts that $\ell_1$ models settle at lower thresholds with many small survivors.

Baselines

Do-nothing references. These are the unconstrained black box and the pure concept bottleneck without a residual. Koh et al. report 17.5% and 19.9% error, respectively, for the black box and the joint bottleneck on CUB (Concept Bottleneck Models). We rerun both on our backbone. These two points bound the frontier: any residual design must land between them in accuracy and should approach the bottleneck's intervention gain.

Strongest existing method. This is HybridCBM, run from its described static and dynamic banks and regularizers. It reports 68.38% on CIFAR-100, against 70.1% for a CLIP linear probe (Hybrid Concept Bottleneck Models). Sparse-CBM reports 80.02% on CUB (Sparse-CBM); we treat it as the strongest CUB reference in the CLIP regime. For CEM we found no reported numeric CUB accuracy in the file, only "comparable to black-box ResNets," so we rerun it with random intervention training.

Fairness and the bar to clear. All methods share the backbone, concept bank, data splits, and a fixed hyperparameter-search budget of equal trials per method. Beating HybridCBM means one of two things. Either we match its accuracy within 0.5 points while achieving clearly higher intervention gain and lower residual support, or we exceed its accuracy at equal intervention gain.

Ablations. Soft ($\ell_1$) and hard straight-through residual gates isolate the operator's shape, since everything else is held fixed. A fixed-threshold versus learned-threshold ablation isolates the benefit of gradient-trained thresholds. The input-adaptive variant is tested against the fixed-threshold firm model.

Why not just…?

Why not just put an $\ell_1$ penalty on the residual? This is the default. Its magnitude bias is documented in Sparse-CBM, in JumpReLU, and in the minimax concave penalty system-identification results. The frontier sweep tests the claim directly. If $\ell_1$ matches firm at every accuracy level, the core claim is false.

Why not just penalize with $\ell_1$, then refit the surviving residual weights without a penalty? This two-stage debiasing idea is the real threat. It removes magnitude bias without a new operator. It fixes the support during refitting, however, and our hypothesis is that the support chosen under $\ell_1$ is already too large. We include it as a baseline. If it matches firm thresholding, we will report that shape matters only through debiasing, and that the practical recommendation is the refit.

Why not just use a hard gate with a straight-through estimator, as in JumpReLU? Hard gates are unbiased. Jumping Ahead notes that the bandwidth setting is sensitive and that dense features appear. We compare variance across random seeds and across the bandwidth grid, as well as frontier position. If the hard gate is equally good and equally stable, our advantage reduces to simplicity.

Why not just train sequentially, fitting concepts first and then the residual on frozen concepts? Sequential training gives the residual only what concepts miss. It does not stop the head from preferring the residual. We run sequential training with every operator, because the two choices are complementary rather than exclusive.

Decisive Checks

The first decisive check is the frontier comparison on CUB with ground-truth interventions. The claim needs firm thresholding to beat both $\ell_1$ and the two-stage refit on intervention gain at matched accuracy. The second is the magnitude-histogram analysis, which must show $\ell_1$ models choosing lower thresholds with more small survivors; otherwise the proposed mechanism is wrong even if the numbers look good. The third is CUB+Places. Koh et al. report roughly 48 to 49% error for concept models against 62.7% for the standard model. A leak-resistant residual should keep robustness near the bottleneck's level, and an $\ell_1$ residual should drift toward the black box. On CIFAR-100, the question is whether the firm residual closes the 1.7-point gap between HybridCBM and the linear probe without raising residual support.