Representer Point Selection via Local Jacobian Expansion for Post-hoc Classifier Explanation of Deep Neural Networks and Ensemble Models

Yi Sui, Ga Wu, Scott SannerNeurIPS 2021 · OpenReview

Introduction

In the quest for "Explainable AI," one powerful technique is Representer Point Selection (RPS). Instead of generating a heatmap of pixels (saliency maps), RPS identifies the specific training examples that most influenced a model's decision. It answers the question: "Why did the model classify this image as a car? Because it learned from these specific training images."

However, the existing state-of-the-art method, RPS-$l_2$, suffers from two major flaws:

  1. Model Mismatch: It requires fine-tuning the model with $l_2$ regularization to work. Consequently, the explanation provided is for a modified version of the model, not the original one deployed in production.
  2. The "Generic" Problem: It often produces nearly identical explanations for every test point in the same class. If you ask why it classified ten different images as "dogs," it might point to the exact same three training images every time, failing to provide instance-specific insight.

To address these limitations, researchers have developed RPS-LJE (Local Jacobian Expansion), a new method that provides individualized and stable explanations without requiring the model to be retrained or modified.

Methods: Local Jacobian Expansion

Unlike previous methods that rely on optimization tricks like $l_2$ regularization, RPS-LJE uses a first-order Taylor expansion on the Jacobian matrix of the model's loss function.

  • Post-hoc Application: It works directly on the already-trained model. No fine-tuning or retraining is required.
  • Theoretical Foundation: It leverages the Representer Theorem to decompose a prediction into a weighted sum of training points. By incorporating second-order information (the Hessian matrix), it accounts for the correlations between model parameters.
  • Broad Compatibility: The method requires the model to have a linear last layer. While this is a standard feature in ResNets and Transformers, RPS-LJE is also remarkably flexible, extending even to gradient-boosted trees like XGBoost.

Datasets and Evaluation

The authors evaluated RPS-LJE across three distinct domains to demonstrate its versatility:

  • Images (CIFAR-10): A binary "horses vs. cars" classification task using a ResNet-20 architecture.
  • Text (IMDB): Sentiment analysis using a Bi-LSTM (Bidirectional Long Short-Term Memory) network.
  • Tabular (German Credit): Credit risk identification using XGBoost, proving the method works for ensemble tree models, not just deep neural networks.

Key Insights and Ideas

Breaking the Class-Level Curse

The authors discovered that in the older RPS-$l_2$ method, the first-order derivative of the loss was so dominant that it drowned out the unique features of the test point. By using the Jacobian expansion, RPS-LJE allows the specific features of the test instance to shine through, leading to truly individualized explanations.

Scatter plots comparing the rank and vector length of explanations. RPS-l2 (left) shows a rigid dependency, while RPS-LJE (right) allows test-instance features to influence the explanation, breaking the 'class-level curse'.

Stability and XGBoost Support

RPS-$l_2$ was highly sensitive to the strength of the $l_2$ penalty—changing the hyperparameter could alter the explanation entirely. RPS-LJE is remarkably stable across different settings, making it a more reliable debugging tool. Furthermore, because RPS-LJE doesn't require fine-tuning (which is notoriously difficult for gradient-boosted trees), it opens up sample-based explanations for a massive class of models used in industry that were previously hard to explain.

Key Results

  • Data Debugging: In a task designed to find mislabeled training data (data curation), RPS-LJE outperformed both the original RPS and Influence Functions. On the CIFAR-10 task, it was 3% more effective at identifying corrupt labels after checking 40% of the data.
  • Qualitative Superiority: Visually, the explanations make much more sense. For a test image of a red truck, RPS-LJE pointed to other red trucks in the training set, whereas the old method often pointed to generic images that appeared in the explanation lists for almost every "car" prediction.
Qualitative results demonstrating RPS-LJE. Unlike previous methods, RPS-LJE provides distinct, individualized training examples (rows) for different test images (columns in block a).
  • Coherence in Text: In text analysis, RPS-LJE identified training examples that shared the same narrative style or sentence structure as the test point, providing a much more intuitive "reasoning" for the model's sentiment prediction.

Limitations and Future Work

While powerful, RPS-LJE has a few limitations:

  • Linear Last Layer: The method assumes the model has a feature embedding stage followed by a linear prediction layer. While this covers most modern architectures, it isn't universal.
  • Saddle Point Assumption: The mathematics assume the model has been trained to near-convergence (a "saddle point" where gradients are near zero). If a model is heavily under-trained or uses extreme early stopping, the Taylor expansion might be less accurate.
  • Computational Overhead: Like many methods involving the Hessian (second-order derivatives), calculating the inverse can be computationally expensive for very large models, though the authors utilize efficient approximations.

Future work could involve extending this logic to models without a clear linear last layer or finding ways to further scale the Hessian calculations for massive "foundation" models.

Summary updated

Abstract

Explaining the influence of training data on deep neural network predictions is a critical tool for debugging models through data curation. A recent tractable and appealing approach for this task was provided via the concept of Representer Point Selection (RPS), i.e. a method the leverages the dual form of $l_2$ regularized optimization in the last layer of the neural network to identify the contribution of training points to the prediction. However, two key drawbacks of RPS are that they (i) lead to disagreement between the originally trained network and the RP regularized network modification and (ii) often yield a static ranking of training data for the same class, independent of the data being classified. Inspired by the RPS approach, we propose an alternative method based on a local Jacobian Taylor expansion (LJE) of the Jacobian. We empirically compared RPS-LJE with the original RPS-$l_2$ on image classification (with ResNet), text classification recurrent neural networks (with Bi-LSTM), and tabular classification (with XGBoost) tasks. Quantitatively, we show that RPS-LJE slightly outperforms RPS-$l_2$ and other state-of-the-art data explanation methods by up to 3\% on a data debugging task. Qualitatively, we observe that RPS-LJE provides individualized explanations for each test data point rather than the class-specific static ranking of points in the original approach. Overall, RPS-LJE represents a novel approach to RPS that provides a powerful tool for data-oriented explanation and debugging.
BibTeX
@inproceedings{sui2021representer,
  author = {Yi Sui and Ga Wu and Scott Sanner},
  title = {{Representer Point Selection via Local Jacobian Expansion for Post-hoc Classifier Explanation of Deep Neural Networks and Ensemble Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=Wl32WBZnSP4},
}
Figure