TL;DR
Concept Bottleneck Models (CBMs) offer interpretable neural classification by predicting high-level concepts before final labels, but they traditionally require expensive human annotations or rely on unreliable Vision-Language Models (VLMs). This paper introduces Vision-plus-Human-guided CBM (VH-CBM), a hybrid approach that uses a Gaussian Process to propagate a tiny amount of expert human supervision (as little as 1%) across a VLM's embedding space. The result is a highly accurate, well-calibrated, and interpretable model that outperforms purely automated VLM-guided alternatives.
What Problem Does This Paper Address?
Concept Bottleneck Models (CBMs) are "explainable-by-design" neural networks. Instead of mapping an input image directly to a class label (a black box), they first predict a set of human-understandable concepts (e.g., "is round", "is red") and use those concepts to make the final prediction.
However, CBMs face a harsh trade-off between interpretability and applicability:
- Standard CBMs require dense, expert-annotated concept labels for the entire training set, making them prohibitively expensive to deploy.
- VLM-guided CBMs circumvent this by using Vision-Language Models (like CLIP) to automatically generate weak concept labels in a zero-shot manner. Unfortunately, VLMs frequently suffer from hallucinations and logical inconsistencies, leading to inaccurate concepts that undermine the model's interpretability.
The authors address this gap by asking: Can we combine the scalability of VLMs with the trustworthiness of human experts without exploding annotation costs?
Methods
The authors introduce VH-CBM, a hybrid framework that treats the VLM not as an automated annotator, but as a continuous geometric space.
The pipeline works as follows:
- Embedding Extraction: Input images are passed through a frozen VLM encoder to obtain continuous embeddings ($z$).
- Gaussian Process Bottleneck: Instead of a standard neural network concept extractor, VH-CBM places a stochastic Gaussian Process (GP) classifier for each concept directly inside the VLM's embedding space.
- Label Propagation: A human expert annotates a very small subset of the data ($D_s$). The GPs leverage the global geometry of the VLM embedding space to smoothly propagate these high-quality human labels to unannotated data points.
- Linear Inference: The concept statistics ($\mu$ and $\sigma$) generated by the GPs are fed into a transparent linear layer to compute the final task prediction.
Because the GPs are Bayesian, they provide reliable predictive uncertainty. The authors exploit this by incorporating an uncertainty-based active learning strategy, prioritizing human annotation for the concepts the model is least confident about.
Datasets
The framework was evaluated across four distinct datasets featuring both concept and task annotations:
- Shapes3d: A synthetic dataset of 3D shapes with 42 binary concept activations (color, shape, etc.).
- CelebA: Real-world celebrity faces annotated with 40 binary attributes (e.g., hair color, makeup).
- CUB (Caltech-UCSD Birds-200-2011): Fine-grained bird images with 112 binary concept activations.
- Derma (DermaMNIST): A challenging medical dataset of skin lesion images with 7 expert-annotated clinical concepts.
Key Insights and Ideas
- Geometry over Point-wise Labels: The core "aha moment" is that while a VLM's zero-shot text-to-image concept matching can be noisy and erratic, its underlying image embedding space is highly structured and globally consistent.
- Bayesian Calibration: By using Gaussian Processes instead of standard deterministic neural network probes, the model achieves near-perfect concept calibration. It knows what it knows, and more importantly, knows when it is uncertain.
- Active Concept Selection: Traditional active learning selects entire images to annotate. VH-CBM allows for active selection at the per-concept level within an image, maximizing the information gain per human interaction.
Key Results
The authors compared VH-CBM (using CLIP and DINO backbones) against fully supervised CBMs and state-of-the-art VLM-CBMs (LaBo, LF-CBM, VLG-CBM) using a strict budget of only 340 annotated concepts.
- Superior Concept Accuracy: As shown in Figure 2, VH-CBM consistently outperforms zero-shot and weakly supervised VLM baselines across datasets, even when using less than 1% of the available training annotations (e.g., on CUB and Shapes3D).
- Near-Zero Calibration Error: VH-CBM achieved an Empirical Cumulative Calibration Error (ECCE) of $\approx 0.02 - 0.04$, drastically outperforming competing VLM-CBMs (some of which scored as high as $0.89$).
- Boosted Task Performance: On fine-grained classification (CUB), VH-CBM outperformed the runner-up VLM-CBM by 21 percentage points in task macro-$F_1$ score, proving that cleaner concepts directly yield better downstream performance.
Limitations and Future Work
While highly effective, VH-CBM has a few limitations noted by the authors:
- Annotation Cost: Though minimal (around 1%), it still requires human-in-the-loop expert intervention, unlike purely automated zero-shot VLM-CBMs.
- GP Scalability: Even though the authors use sparse GPs with inducing points to scale linearly with the number of concepts, training independent GPs for hundreds or thousands of concepts can still introduce computational overhead.
- Concept Entanglement: While concept accuracy is high, explicitly enforcing concept disentanglement (ensuring concepts don't leak information about each other) remains an open challenge for future work.
Summary updated