A concept bottleneck model is a diagnostic network that first predicts human-readable clinical findings, such as "irregular streaks" or "spiculated margin," and then makes its diagnosis from those findings alone. It is attractive for medicine because a doctor can inspect and correct the findings. It has two well-known failures. First, the predicted findings quietly carry extra information that is not about the findings, and the diagnosis relies on it. This is called leakage. Second, forcing a network to predict findings can hurt diagnosis on cases the findings do not describe well. We propose a different construction. Take a frozen pretrained image encoder and learn an exactly invertible map from its feature vector to a new coordinate system. A few coordinates are trained to be the clinical concepts, and the remaining coordinates hold everything else. Because the map loses no information, three things become possible. We can check exactly how much the diagnosis depends on non-concept information. Editing a concept and mapping back yields an exact counterfactual feature vector, with no approximating generator involved. And the concept head cannot take capacity away from the rest of the representation.
The setting is small cohorts, roughly 50 to 1,000 labeled patients, where only a small head can be trained. That constraint shapes the design. Results on deep narrow networks say that general smooth maps need hidden layers wider than the input. Invertible maps can instead be built at exactly the input width, from coupling layers or from rotations interleaved with learnable one-dimensional monotone warps. We treat this as a ladder of expressiveness at very different parameter counts. A central experimental question is where on that ladder small clinical datasets stop benefiting.
Core Claim
Let $h \in \mathbb{R}^d$ be the frozen encoder's feature for an image. We learn a bijection $T:\mathbb{R}^d \to \mathbb{R}^k \times \mathbb{R}^{d-k}$, written $T(h) = (c, r)$. Here $c$ holds $k$ concept coordinates supervised by clinical annotations, and $r$ is the residual. A diagnosis head $f$ reads only $c$. Training minimizes
$$ \mathcal{L} = \ell_{y}\big(f(c), y\big) + \lambda \sum_{j=1}^{k} \ell_{c}\big(c_j, a_j\big) + \mu\, \widehat{I}\big(r; a\big), $$
where $a_j$ is the annotated value of concept $j$, $\ell_y$ and $\ell_c$ are cross-entropy or regression losses, and $\widehat{I}(r;a)$ is an estimate of how predictable the annotated concepts are from the residual. That last term is implemented adversarially by a small probe. It pushes concept information into $c$ and out of $r$.
The claim is not that invertibility raises accuracy. It is that invertibility makes three measurements exact rather than approximate.
The first is leakage. Since $(c, r)$ contains all of $h$, we can quantify how much of the diagnosis $f(c)$ explains beyond what hard, binarized concepts explain. We can also quantify how much a probe on $r$ adds.
The second is counterfactuals. For an edited concept vector $c'$, the feature $h' = T^{-1}(c', r)$ is a real point in the encoder's feature space. Any independent classifier trained on $h$ can therefore be used to check whether the concept explanation agrees with what the features imply.
The third is capacity. Because $T$ discards nothing, the representation available to other readers is unchanged. The per-class accuracy drop reported by Veltmeijer et al. should therefore not appear in a black-box probe on $T(h)$. When it does appear in $f(c)$, it can be attributed to concept incompleteness rather than representational interference.
Positioning
Three lines of work motivate the proposal. Concept Bottleneck Models introduced the concept-then-diagnosis structure. They showed test-time concept correction on knee X-rays (fixing two concepts lowered error from above 0.4 to about 0.3) and robustness to background confounders on CUB+Places. Their bottleneck, however, is a lossy network layer, which is where leakage lives. Yuksekgonul et al. removed the annotation burden by scoring frozen vision-language features against generated concept text and training a linear classifier on top. This is the closest small-sample design, and it motivates our use of frozen features. Their concept map, though, is a projection and cannot be inverted. CCBM adds unsupervised "complement" concepts for incomplete vocabularies. Its authors report that these complement concepts cannot be interpreted and that training is unstable on the small BrEaST cohort. Our residual $r$ plays the complement's role. The difference is that $r$ is exactly decodable back to feature space, so it can be inspected through nearest neighbors, rather than being an opaque adapter output.
For counterfactuals, Counterfactual CBMs generate "why not class $y'$" concept vectors with a variational encoder. That encoder only approximates the data. We replace it with an exact inverse and test whether exactness improves counterfactual validity when data are scarce, where variational models are least reliable. ProbCBM models concept uncertainty. That idea is compatible with ours, but we do not pursue it here.
The method comes from the expressivity literature. Ishikawa et al. prove that affine coupling layers interleaved with invertible linear maps approximate smooth diffeomorphisms in $L^p$ without increasing dimension. This is what licenses keeping $T$ at width $d$. By contrast, Hwang's minimum-width result shows that a narrow multilayer perceptron needs width at least $\min(n+m, \max(2n+1, m))$ to approximate general $C^1$ maps. This explains why a non-invertible encoder of the same width is not an equivalent alternative. DiTAC supplies monotone one-dimensional diffeomorphic activations at 9 extra parameters each. These are the cheap nonlinearity in our middle rung.
LU-Net constrains the design: it reports that fixed-width bijective networks are hard to optimize. We mitigate this by acting on frozen features rather than pixels, keeping depth small, and initializing $T$ at the linear solution. We treat optimization failure as a measurable outcome, not something to hide.
Research Program
We build $T$ at three rungs.
The first rung is an orthogonal rotation. It is initialized by aligning the leading coordinates with concept directions found by linear regression on concept labels, and has about $d^2/2$ free parameters.
The second rung stacks rotations interleaved with per-coordinate DiTAC-style monotone warps. It is invertible at width $d$ with only 9 parameters per coordinate per layer. Whether this class is universal for diffeomorphisms is, as far as the file shows, not established. We will study it empirically and state that honestly rather than claim it.
The third rung is affine coupling flows with 1×1 mixing, which carry the Ishikawa guarantee.
The diagnosis head $f$ is linear on $c$ by default. An additive per-concept-spline variant is used where concept interactions matter.
We analyze three quantities.
The first is leakage. It has two parts. One is the gap between accuracy of $f$ on soft $c$ and accuracy on binarized $c$. The other is the accuracy gain from appending $r$ to the head, measured with a held-out probe.
The second is round-trip faithfulness. We edit concept $j$ in $c$, decode $h'=T^{-1}(c',r)$, and measure agreement in the direction and size of prediction change between $f$ and an independent black-box classifier on $h$.
The third is intervention efficiency. We correct the $m$ most uncertain concepts to their annotated values and record the resulting task error as a function of $m$.
Baselines
The naive reference is a logistic or linear probe on the frozen feature $h$, with no concepts. It is the black-box upper bound for diagnosis accuracy at each sample size, and it is what the capacity claim is measured against. It gets the same features, the same splits and the same hyperparameter search.
The strongest small-sample interpretable competitor is the label-free linear concept classifier of Yuksekgonul et al.. We run it on the same frozen encoder features with a linear head. On the file's confounded chest X-ray tasks it reports 50.7% accuracy on NIH-age, against 9.4% for raw BioViL features. On Derm7pt we use annotated concepts for both it and our method, so the comparison isolates the mapping and not the concept source.
The strongest concept-and-diagnosis method on our main benchmark is CCBM. It reports a concept detection AUC of 93.96% on Derm7pt. For diagnosis on Derm7pt, the best figure in the file is SkinM2Former at 77.85% mean diagnostic accuracy. SkinM2Former is not a bottleneck model, so it serves as an accuracy reference point only. We rerun CCBM from its description on our frozen backbone at matched head-parameter budgets and tuning trials. Beating it means concept AUC within one point, diagnosis within one point, and lower leakage at every $N \le 250$. On BrEaST we found no reported result on Lacuna for CCBM or any other method, so all BrEaST numbers will be our own reruns.
A joint concept bottleneck model in the style of Koh et al. is also included, with an MLP encoder from $h$ to $c$. For counterfactual validity, CF-CBM is the reference, trained on the same features.
Two ablations isolate the key ingredient. The first is a non-invertible MLP from $h$ to $(c, r)$ with a matched parameter count and the same loss. It keeps everything except bijectivity. If it matches our method on leakage and round-trip faithfulness, invertibility is not doing the work. The second ablation compares the three rungs against each other.
Why not just…?
Why not just use an orthogonal rotation of the features? An orthogonal rotation is exactly invertible, has no optimization pathology, and may well be sufficient. That makes it the most serious threat, so it is our first rung and not a strawman. The thing it cannot do is make a concept that depends nonlinearly on features into a single coordinate. When that happens, concept information spills into $r$ and the leakage probe on $r$ detects it. The experiment settles the question directly: if the rotation matches the warped and coupling rungs on concept AUC and residual leakage at all $N$, we will report that linear is enough. The invertibility-based audit and exact counterfactuals remain the contribution in that case.
Why not just add a decorrelation penalty to a standard concept bottleneck with a side channel? A decorrelation penalty can reduce measured leakage. But the bottleneck itself still discards information, so there is no exact inverse for counterfactuals, and no guarantee that "concept" and "residual" together account for the representation. The matched non-invertible MLP ablation carries exactly this penalty, and it tests whether bijectivity adds anything beyond the loss term.
Why not just use CF-CBM's learned counterfactual generator? CF-CBM's generator is variational, so its counterfactuals are only as good as its approximation of the data. It was evaluated with a single medical benchmark in the file, on pneumothorax. At $N=50$ to $250$ we expect a learned generator to drift, whereas an exact inverse cannot. Round-trip faithfulness against an independent classifier measures this directly. If CF-CBM matches us at small $N$, the exactness argument fails.
Why not just fine-tune a black box and explain it with saliency maps? Saliency maps do not support concept correction. Koh et al.'s intervention results and CF-CBM's comparison against pixel-level counterfactual search both indicate that post-hoc explanations are weaker in the respects that matter here. We keep the black-box probe as the accuracy reference and do not compete with it on accuracy.
Proposed Experiments and Decisive Checks
The main benchmark is Derm7pt, with 1,011 cases and seven-point checklist concepts. We subsample training sets to $N$ = 50, 100, 250 and the full set, using ten seeds per size. BrEaST supplies a genuinely small cohort with BI-RADS descriptor concepts. LIDC-IDRI is used with radiologist attributes, and a secondary experiment trains the diagnosis head on the biopsy-grounded labels studied by Zhang et al.. That work reports a sensitivity/specificity shift from 0.97/0.23 to 0.67/0.66 after relabeling. We add ISIC 2017 per-class analysis to revisit the nevus degradation reported by Veltmeijer et al. All methods share one frozen encoder per modality.
We report concept AUC, diagnosis accuracy and AUC, per-class accuracy, the two leakage measures, round-trip faithfulness, and intervention curves.
Three results would be decisive.
The first concerns leakage. If our method's residual-probe gain and soft-versus-hard gap are not lower than those of the matched non-invertible MLP and joint CBM at $N \le 250$, the core claim fails.
The second concerns counterfactuals. If round-trip agreement for exact inverses does not exceed CF-CBM's at small $N$, exact counterfactuals offer no practical advantage.
The third concerns nevus degradation. If the black-box probe on $T(h)$ shows the nevus-class drop that the concept head shows, the representation itself is interfering, and the capacity argument is wrong. If only $f(c)$ shows the drop, the vocabulary is incomplete. In that case, decoding the residual directions that separate nevi, via nearest neighbors in feature space, becomes a concrete tool for proposing missing concepts to clinicians.
Across rungs, we expect the gap between the rotation and the coupling flows to widen with $N$. The warped rung should win at the smallest sizes. Any other pattern will be reported as found.
References
- Integrating Clinically-Relevant Features into Skin Lesion Classification
- Concept Bottleneck Models
- Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models
- Concept Complement Bottleneck Model for Interpretable Medical Image Diagnosis
- Counterfactual Concept Bottleneck Models
- Probabilistic Concept Bottleneck Models
- Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
- Optimal Minimum Width for the Universal Approximation of Continuously Differentiable Functions by Deep Narrow MLPs
- Trainable Highly-Expressive Activation Functions
- LU-Net: Invertible Neural Networks Based on Matrix Factorization
- A Novel Perspective for Multi-Modal Multi-Label Skin Lesion Classification
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction