SOInter: A Novel Deep Energy-Based Interpretation Method for Explaining Structured Output Models

S. Fatemeh Seyyedsalehi, Mahdieh Soleymani Baghshah, Hamid R. RabieeICLR 2024 · OpenReview

TL;DR

SOInter is a novel post-hoc interpretation method specifically designed to explain "structured output models" (like image segmentation or multi-label classification) where output variables are highly correlated. By utilizing an energy-based training process, SOInter accounts for these complex inter-variable dependencies to identify the most influential input features for a specific target output.

What Problem Does This Paper Address?

Most popular explainability (XAI) tools, such as LIME or SHAP, treat output variables as independent entities. However, many real-world machine learning tasks involve structured outputs—where the value of one output depends on others. For example, in image segmentation, the classification of one pixel is heavily influenced by its neighbors. In multi-label text classification, certain tags (like "physics" and "quantum") often appear together.

In these models, a feature might influence a target output variable indirectly through its relationship with other output variables. Standard interpretation methods often miss these "structural" nuances, leading to explanations that are incomplete or fail to capture the model's true decision-making logic. The authors address this gap by proposing a method that explicitly incorporates these structural correlations into the explanation process.

Methods

The authors introduce SOInter (Structured Output Interpreter), which treats the model to be explained as a black box. The core architecture consists of two main components:

  1. The Interpreter Block: A neural network that learns a mapping from the input space to a $k$-hot vector. This vector identifies the $k$ most important features for a specific target output. To make this discrete selection process differentiable, the authors employ the Gumbel-Softmax trick.
  2. The Energy Block: This is the "secret sauce" of the paper. The authors train a deep energy-based model to represent the consistency between inputs and outputs of the black-box model.

The Training Process: The system is trained using a novel loss function based on energy differences. Essentially, if the interpreter correctly identifies the "important" features, the energy of the model's prediction should remain low even if the "unimportant" features are perturbed.

The training happens in two phases:

  • Pre-training: The energy block is trained to mimic the black-box model's behavior.
  • Fine-tuning: The interpreter and the energy block are updated iteratively. This ensures the energy block remains accurate even in the specific regions of the feature space that the interpreter focuses on.

Datasets

The effectiveness of SOInter was evaluated across three distinct types of data:

  • Synthetic Datasets: Created using linear and non-linear energy functions to provide a "ground truth" for feature importance, allowing for precise accuracy measurements.
  • Multi-label Text Classification: Three standard benchmarks—Bibtex (159 tags), Enron (53 topics), and IMDB (28 genres).
  • Image Segmentation: The Weizmann-horses dataset, which consists of 328 images and masks used to train a deep value network for segmentation.

Key Insights and Ideas

  • Structure is a First-Class Citizen: The primary insight is that you cannot accurately explain a structured model by looking at outputs in isolation. Incorporating the "structural information" (correlations between outputs) significantly reduces uncertainty about the black-box's behavior.
  • Energy as a Proxy for Logic: By training an energy-based model to act as a surrogate for the black box, the authors create a differentiable landscape that the interpreter can use to "search" for important features.
  • Global Training for Local Explanations: Unlike perturbation-based methods that require hundreds of model passes for a single explanation, SOInter is trained globally. Once trained, it can generate a local explanation for a new sample in a single forward pass.

Key Results

  • Higher Accuracy: On synthetic data, SOInter consistently outperformed LIME, Kernel-SHAP, and L2X, achieving the nearest median rank to the ground-truth important features.
Accuracy Comparison
Median Rank Comparison
  • Better Faithfulness: In text classification tasks, SOInter achieved higher "post-hoc F1" scores. This means that when the model was given only the features SOInter identified as important, it was much better at reconstructing its original prediction compared to when it used features selected by other methods.
  • Scalability: SOInter demonstrates constant explanation time regardless of the input vector size. In contrast, the computational cost of LIME and Kernel-SHAP grows significantly as the number of features increases, making SOInter much better suited for high-dimensional data.
  • Qualitative Success: In image segmentation, SOInter correctly identified input pixels in the immediate semantic neighborhood of the target pixel, confirming it captures local spatial dependencies.

Limitations and Future Work

  • Feature Corruption: Like many perturbation-based methods, SOInter "removes" features by setting them to zero. The authors acknowledge that if "zero" has a specific meaning to the black-box model, this might lead to misleading results.
  • Independent Selection: Currently, the model selects individual features (e.g., single pixels or words). The authors suggest that future work could extend the method to detect "coarse conceptual elements"—groups of features that function together as a single concept.
  • Memory Overhead: Because SOInter requires maintaining and training two additional neural networks (the interpreter and the energy block), it has a higher memory footprint during the training phase than simpler, non-learning-based explainers.

Summary updated

Abstract

This paper proposes a novel interpretation technique to explain the behavior of structured output models, which simultaneously learn mappings between an input vector and a set of output variables. As a result of the complex relationships between the computational path of output variables in structured models, a feature may impact an output value via other output variables. We focus on one of the outputs as the target and try to find the most important features adopted by the structured model to decide on the target in each locality of the input space. We consider an arbitrary structured output model available as a black-box and argue that considering correlations among output variables can improve explanation quality. The goal is to train a function as an interpreter for the target output variable over the input space. We introduce an energy-based training process for the interpreter function, which effectively considers the structural information incorporated into the model to be explained. The proposed method's effectiveness is confirmed using various simulated and real data sets.
BibTeX
@inproceedings{seyyedsalehi2024sointer,
  author = {S. Fatemeh Seyyedsalehi and Mahdieh Soleymani Baghshah and Hamid R. Rabiee},
  title = {{SOInter: A Novel Deep Energy-Based Interpretation Method for Explaining Structured Output Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=Fn655mJ4bv},
}
Figure