Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat

Shantanu Ghosh, Ke Yu, Forough Arabshahi, Kayhan Batmanghelich

TL;DR

This paper introduces "Route, Interpret, Repeat," a novel framework that iteratively decomposes a high-performing blackbox model into a Mixture of Interpretable Experts (MoIE). By carving out symbolic logic explanations for specific data subsets while routing "harder" samples to a residual blackbox, the authors achieve state-of-the-art performance without sacrificing the transparency of inherently interpretable models.

What Problem Does This Paper Address?

In machine learning, there is a long-standing dichotomy: you can either use a flexible, high-performing Blackbox (like a deep neural network) and try to explain it after the fact (post hoc), or you can build an Interpretable-by-Design model from the start.

Both have major drawbacks. Post hoc explanations often lack fidelity to the original model and can be prone to confirmation bias. Conversely, interpretable-by-design models (like Concept Bottleneck Models) often underperform because they assume a single, simple function can explain an entire complex dataset. This paper bridges the gap, aiming to retain blackbox-level accuracy while providing the mechanistic, actionable explanations typical of symbolic models.

Methods

The authors propose an iterative "divide-and-conquer" approach called Route, Interpret, Repeat. Instead of trying to explain the whole model at once, they break it down into a sequence of experts.

  1. Route: A learnable gating mechanism (a "selector") examines an input and decides if it can be explained by the current interpretable expert.
  2. Interpret: If selected, the sample is processed by a Neuro-Symbolic Interpretable Expert. This expert uses First Order Logic (FOL) to provide reasoning based on human-understandable concepts (e.g., "If the bird has a black forehead AND grey legs, then it is a Bay Breasted Warbler").
  3. Repeat: Samples that the current expert cannot explain are routed to a Residual Blackbox. This residual is then used as the starting point for the next iteration, where a new expert is "carved out."
Figure 1. Schematic view of route, interpret and repeat. At it- eration k, the selector routes each sample either towards the in- terpretable model gk (to interpret) with probability πk(.) or the residual rk = f k−1 −gk with probability 1 −πk(.) (to repeat in the further iterations). f k−1 is the Blackbox of the (k −1)th

This process continues until a desired proportion of the data is covered by interpretable experts. The final model is a Mixture of Interpretable Experts (MoIE) plus a small residual for the most "unexplainable" samples.

Datasets

The framework was evaluated across diverse vision and medical imaging tasks:

  • CUB-200 & AwA2: Standard benchmarks for fine-grained bird and animal classification.
  • HAM10000 & SIIM-ISIC: Medical datasets for skin lesion and melanoma classification.
  • MIMIC-CXR: A large-scale chest x-ray dataset used here for identifying pleural effusion.
  • Waterbirds: A synthetic dataset specifically designed to test "shortcut learning" (e.g., a model mistakenly identifying a bird based on the background rather than the bird itself).

Key Insights and Ideas

  • Blackboxes are Multi-Faceted: The core hypothesis is that a single blackbox actually encodes multiple different interpretable strategies. MoIE succeeds by identifying these distinct strategies and assigning them to specialized experts.
  • Instance-Specific Explanations: Unlike standard models that give a generic explanation for a whole class, MoIE identifies diverse, instance-specific concepts. For example, one expert might identify a bird by its wing color, while another focuses on its beak shape.
  • Identifying "Hard" Samples: The iterative process naturally sorts data by difficulty. Samples that reach the final residuals are quantitatively shown to be the "hardest" for the original blackbox to classify correctly.
  • Fixing Shortcuts with Logic: Because the experts use First Order Logic, humans can intervene. If an expert is found to be using a "shortcut" (like a forest background to identify a land bird), the logic can be modified or the bias can be removed using metadata normalization.
Figure 2. MoIE identifies diverse concepts for specific subsets of a class, unlike the generic ones by the baselines. (i) We construct the FOL explanations of the samples of, “Bay breasted warbler” in the CUB-200 dataset for VIT-based (a) CBM + E-LEN as an interpretable-by-design baseline, (b) PCBM + E-LEN as a posthoc baseline, (c) experts in MoIE at inference. We highlight the unique concepts for experts 1,2, and 3 in red, blue, and magenta, respectively. (ii) Comparison of FOL explanations by MoIE with the PCBM + E-LEN baselines for HAM10000 (top) and ISIC (down) to classify Malignant lesion. We highlight unique concepts for experts 3, 5, and 6 in red, blue, and violet, respectively. For brevity, we combine the local FOLs for each expert for the samples covered by them, shown in the figure.

Key Results

  • Performance Parity: MoIE achieves accuracy nearly identical to the original blackbox models, significantly outperforming traditional interpretable-by-design baselines like CBMs. For instance, on CUB-200 (VIT), MoIE reached 91% accuracy compared to the blackbox's 92%, while the CBM baseline lagged at 86%.
  • Higher Fidelity: MoIE-identified concepts attained roughly 20% higher completeness scores than post-hoc baselines, meaning the concepts chosen were much more essential to the model's actual decision-making process.
  • Effective Intervention: In "test-time interventions"—where a human corrects a misidentified concept—MoIE showed superior recovery. On CUB-200, intervening on just 20 concepts boosted MoIE accuracy from 91% to 96%.
  • Shortcut Removal: The authors successfully used MoIE to identify and eliminate spurious background biases in the Waterbirds dataset, creating a more robust model that generalizes better across different environments.
MoIE successfully identifies and fixes shortcut learning (background bias)

Limitations and Future Work

  • Causality: The authors acknowledge that while the identified concepts are highly predictive, they are not necessarily causal. Future work will focus on estimating inter-concept interactions to better assess causal effects.
  • Modality Expansion: The current study focuses on images. The researchers aim to apply the "Route, Interpret, Repeat" framework to other data types, such as text and video, in the future.
  • Computational Overhead: While experts are shallow and efficient, the iterative training process adds some complexity compared to training a single standard model.

Summary updated

Abstract

ML model design either starts with an interpretable model or a Blackbox and explains it post hoc. Blackbox models are flexible but difficult to explain, while interpretable models are inherently explainable. Yet, interpretable models require extensive ML knowledge and tend to be less flexible and underperforming than their Blackbox variants. This paper aims to blur the distinction between a post hoc explanation of a Blackbox and constructing interpretable models. Beginning with a Blackbox, we iteratively carve out a mixture of interpretable experts (MoIE) and a residual network. Each interpretable model specializes in a subset of samples and explains them using First Order Logic (FOL), providing basic reasoning on concepts from the Blackbox. We route the remaining samples through a flexible residual. We repeat the method on the residual network until all the interpretable models explain the desired proportion of data. Our extensive experiments show that our route, interpret, and repeat approach (1) identifies a diverse set of instance-specific concepts with high concept completeness via MoIE without compromising in performance, (2) identifies the relatively ``harder'' samples to explain via residuals, (3) outperforms the interpretable by-design models by significant margins during test-time interventions, and (4) fixes the shortcut learned by the original Blackbox. The code for MoIE is publicly available at: \url{https://github.com/batmanlab/ICML-2023-Route-interpret-repeat}
BibTeX
@inproceedings{ghosh2023dividing,
  author = {Shantanu Ghosh and Ke Yu and Forough Arabshahi and Kayhan Batmanghelich},
  title = {{Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=0SgBUsL4W0},
}
Figure