Interpreting Deep Neural Networks with the Package innsight

Niklas Koenen, Marvin N. Wright

Deep neural networks are famously powerful but notoriously opaque. For years, if you wanted to understand why a model predicted a specific outcome—a field known as Explainable AI (XAI)—you likely had to leave the R ecosystem and head over to Python. While bridges like reticulate exist, they often come with a "Python tax" of complex dependencies and environment management.

In their paper, "Interpreting Deep Neural Networks with the Package innsight," Niklas Koenen and Marvin N. Wright introduce a game-changer for the R community: a native, high-performance, and library-agnostic toolbox for deep learning interpretability.

TL;DR

The innsight package is the first unified R framework for feature attribution in neural networks. It allows researchers to peek inside "black box" models from keras, torch, or neuralnet using a fast C++ backend (LibTorch) to provide clear, interactive visualizations of which input variables drive a model's predictions.

What Problem Does This Paper Address?

As neural networks move into high-stakes domains like healthcare and autonomous systems, "because the model said so" is no longer an acceptable answer. We need feature attribution methods—techniques that assign a "relevance score" to each input (like a specific pixel in an image or a variable in a spreadsheet) to show its contribution to the final prediction.

Until now, the most popular tools for this (like captum or innvestigate) were Python-exclusive. R users were stuck between a rock and a hard place: either use simpler, slower "model-agnostic" methods that don't look at the network's internal structure, or deal with the headache of cross-language integration. innsight fills this gap by providing a native R solution that is both deep-learning-library-agnostic and computationally efficient.

Methods

The authors implement a "3-step workflow" that standardizes the XAI process:

  1. Convert: A convert() function takes a trained model (from Keras, Torch, etc.) and replicates it into an internal torch (R version) structure.
  2. Apply: Users choose from a suite of state-of-the-art attribution methods.
  3. Visualize: The package generates high-quality ggplot2 or interactive plotly charts.

Key algorithms included: * Gradient-based: Vanilla Gradients, SmoothGrad, Integrated Gradient, and Expected Gradient. * Rule-based: Layer-wise Relevance Propagation (LRP) with various rules (simple, $\epsilon$, $\alpha$-$\beta$) and DeepLift. * Classic/Agnostic: Connection Weights (CW), plus wrappers for LIME and KernelSHAP.

What makes innsight unique is its use of the R torch package. By building on LibTorch (the C++ backend of PyTorch), it performs heavy-duty array calculations without ever needing a Python installation on the user's machine.

Datasets

The authors validated innsight using three distinct scenarios:

  • Bike Sharing Dataset: A regression task used to demonstrate tabular data interpretation.
  • Palmer Penguins: A classification task used to show how the model distinguishes between species based on physical traits.
  • SIIM-ISIC Melanoma Dataset: A complex, multimodal dataset (images of skin lesions + tabular patient data). This was used to showcase the package's ability to handle high-dimensional image data and combined inputs.

Key Insights and Ideas

  • The "Universal Translator" for Models: The most clever part of innsight is its ability to ingest models from different libraries. By converting them into a unified internal format, it ensures that the same XAI method can be applied to a Keras model and a neuralnet model with identical syntax.
  • Simultaneous Multi-Output Calculation: Unlike many Python counterparts that calculate attributions for one output class at a time, innsight is optimized to calculate relevances for multiple output nodes simultaneously, significantly speeding up the process for multi-class problems.
  • Interactive XAI: By integrating with plotly, the package allows users to hover over pixels or bars to see exact relevance values, making the "explanation" much more exploratory and user-friendly.

Key Results

  • Numerical Precision: The authors compared innsight against five major Python libraries. The results were nearly identical, with mean absolute differences staying below $10^{-6}$ (the threshold for floating-point errors), proving that the R implementation is as reliable as the industry standards.
  • Performance: innsight proved to be significantly faster than Keras-based Python tools (innvestigate, deeplift). While slightly slower than the highly optimized captum (PyTorch) for some tasks, it was competitive and sometimes faster when dealing with multiple output nodes.
  • Multimodal Success: In the melanoma example, innsight successfully highlighted the "ragged borders" of malignant lesions as high-relevance areas, aligning with how human dermatologists actually diagnose skin cancer.

Limitations and Future Work

While a massive step forward, the authors note a few current boundaries:

  • Sequential Only: Currently, the automatic converter primarily supports sequential models. Complex, non-sequential nn_modules from the R torch package require manual list-based conversion.
  • CPU Bound: The current version of innsight supports CPU calculations only; GPU support is a target for future updates.
  • Method Expansion: The authors plan to add more "perturbation-based" methods like Occlusion or RISE in future iterations.

For R users in data science and bioinformatics, innsight represents the end of "Python envy" for model interpretability. It’s a robust, fast, and deeply integrated tool that finally brings the "why" of deep learning to the R console.

Summary updated

Abstract

The R package innsight offers a general toolbox for revealing variable-wise interpretations of deep neural networks' predictions with so-called feature attribution methods. Aside from the unified and user-friendly framework, the package stands out in three ways: It is generally the first R package implementing feature attribution methods for neural networks. Secondly, it operates independently of the deep learning library allowing the interpretation of models from any R package, including keras, torch, neuralnet, and even custom models. Despite its flexibility, innsight benefits internally from the torch package's fast and efficient array calculations, which builds on LibTorch $-$ PyTorch's C++ backend $-$ without a Python dependency. Finally, it offers a variety of visualization tools for tabular, signal, image data or a combination of these. Additionally, the plots can be rendered interactively using the plotly package.
BibTeX
@article{koenen2024interpreting,
  author = {Niklas Koenen and Marvin N. Wright},
  title = {{Interpreting Deep Neural Networks with the Package innsight}},
  journal = {J. Stat. Softw.},
  year = {2024},
  doi = {10.18637/jss.v111.i08},
}

Loading related content...

Figure