Cutting the Black Box: Conceptual Interpretation of a Deep Neural Net with Multi-Modal Embeddings and Multi-Criteria Decision Aid

Nicolas Atienza, Roman Bresson, Cyriaque Rousselot, Philippe Caillou, Johanne Cohen, Christophe Labreuche, Michèle Sebag

Introduction

Deep Neural Networks (DNNs) have revolutionized computer vision but remain largely opaque "black boxes." While saliency maps indicate where a model focuses, they fail to explain what high-level concepts drive a prediction. Furthermore, existing Concept Activation Vector (CAV) methods are labor-intensive, requiring manual labeling of thousands of examples. The Cut the Black Box (CB2) framework addresses these challenges by distilling complex teacher models into transparent student models that utilize human-understandable logic.

Methodology: The CB2 Framework

The CB2 framework employs a three-stage process to achieve "transparent-by-design" interpretability:

  1. Conceptual Representation: Images are mapped into a vector of concept scores using CLIP (Contrastive Language-Image Pre-training). This aligns visual data with a text-based dictionary of concepts (e.g., "wings," "wheels") in a shared embedding space.
  2. MCDA Student Distillation: A student model is trained to mimic a pre-trained teacher (e.g., ResNet or VGG). Unlike standard neural networks, this student uses Multi-Criteria Decision Aid (MCDA)—specifically, 2-additive Hierarchical Choquet Integrals (2-HCI). This mathematical structure allows for explicit observation of how concepts are weighted and combined.
  3. Latent Alignment: An auto-encoder aligns the teacher’s internal latent representation with the student’s conceptual representation, ensuring the explanation reflects the teacher's actual internal logic.
The CB2 framework architecture, illustrating the distillation of a Teacher model into a transparent Student model using CLIP embeddings and latent alignment.

To quantify the contribution of each concept, the framework uses Winter values (a game-theoretic variation of Shapley values) to rank concept importance for specific classifications.

Experimental Setup

The framework was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, Tiny ImageNet, and Fashion-MNIST. Two types of concept dictionaries were tested: Common Knowledge (CK) derived from Large Language Models and COCA (general English nouns/adjectives).

Results and Analysis

The CB2 framework demonstrates high faithfulness to the teacher models. On CIFAR-10, the student achieved 91% accuracy, slightly surpassing its ResNet-18 teacher (89%).

Decision Sparsity and Concept Importance

A key finding is "decision sparsity": complex classifications often rely on a few dominant concepts. For instance, the "truck" class is largely explained by concepts like delivery branding and truck cab.

Winter values for various CIFAR-10 classes, showing the relative importance of specific concepts and the 'residual' unexplained logic.

The inclusion of a "residual" factor is a critical feature of CB2. It quantifies the portion of the decision not captured by the current dictionary, providing a transparent measure of the explanation's completeness.

Exposing Model Bias

CB2 can reveal "shallow" reasoning or dataset biases. For example, if a "deer" is classified primarily based on the concept of "grazing," the model may fail in environments without grass. This visibility allows researchers to identify and mitigate potential failures before deployment.

Conclusion and Future Work

While CB2 offers significant transparency, it is currently limited by its dependence on the provided dictionary and the inherent biases of the CLIP model used for grounding. Future research will focus on autonomous dictionary discovery, enabling the system to identify relevant concepts without human intervention, and improving scalability for datasets with thousands of classes.

Summary updated

BibTeX
@inproceedings{atienza2024cutting,
  author = {Nicolas Atienza and Roman Bresson and Cyriaque Rousselot and Philippe Caillou and Johanne Cohen and Christophe Labreuche and Michèle Sebag},
  title = {{Cutting the Black Box: Conceptual Interpretation of a Deep Neural Net with Multi-Modal Embeddings and Multi-Criteria Decision Aid}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure