TL;DR
This paper provides a rigorous mathematical foundation for Layer-wise Relevance Propagation (LRP), proving it is not just a heuristic but the solution to a variational problem rooted in entropic optimal transport. By formalizing neural networks as graphs of Filippov operators, the authors extend attribution methods to modern, discontinuous architectures like hard-routed Mixture-of-Experts (MoE) and introduce a new, highly faithful rule called Sinkhorn Relevance Propagation.
What Problem Does This Paper Address?
Explainable AI (XAI) aims to peek inside the "black box" of neural networks to understand why they make specific predictions. Two popular approaches are gradient-based methods (sensitivity) and relevance propagation (attribution). While relevance propagation (specifically LRP) is widely used, it has long been criticized as a heuristic based on local conservation laws rather than a formal optimization principle.
Furthermore, standard XAI tools struggle with modern architectures that aren't smooth. For example, hard-routed Mixture-of-Experts (MoE) models—increasingly common in large language models—feature discrete switching logic that creates discontinuities. At these "routing boundaries," classical gradients are undefined, making traditional attribution methods unreliable or mathematically unsound.
Methods
The authors bridge these gaps using several novel techniques:
- Filippov Neural Networks: They redefine feed-forward networks as directed acyclic graphs (DAGs) of Filippov operators. This measure-theoretic approach allows for a generalized-gradient calculus that remains well-defined even when the network map jumps or is non-differentiable.
- Entropic Optimal Transport (EOT): The core contribution is showing that LRP is exactly the entropic projection of a "backward transition law" onto the set of divergence-free flows. In this view, different LRP rules ($LRP_0, LRP_{z+}, LRP_{\alpha,\beta}$) simply correspond to different choices of a "reference measure."
- Sinkhorn Relevance Propagation (SRP): By applying entropy edge-by-edge rather than across transition laws, the authors derive a new attribution rule computed via Sinkhorn-type iterations.
- Schrödinger Bridges: To explain specific regions of an image (e.g., "why did the model look at the dog's head?"), they use the framework of Schrödinger bridges to condition the relevance flow on specific input subsets.
Datasets
The authors evaluated their framework on standard image classification benchmarks:
- ImageNet: Used to test VGG-16 and ConvNeXt-Tiny architectures.
- Imagenette: A smaller 10-class subset of ImageNet used for detailed faithfulness metrics (ABPC).
- Oxford-IIIT Pet: Used for region-based relevance evaluation, utilizing the dataset's animal "trimaps" and head bounding boxes.
Key Insights and Ideas
- LRP is Variational: The "aha" moment is that LRP isn't just a redistribution trick; it’s finding the "most likely" flow of information that satisfies conservation constraints, where "most likely" is defined by minimizing relative entropy to the forward pass.
- Backprop as Optimal Control: The authors show that standard backpropagation can also be characterized variationally as the adjoint equation of a Lagrangian relaxation in a discrete-time optimal control problem.
- Hebbian Relevance: They introduce a new rule, $LRPN_{\alpha,\beta}$, where weights vary based on node activity. This follows a Hebbian-inspired principle: more active nodes send more relevance to their excitatory contributions.
- Unified DAG Framework: Their formalization covers everything from simple ReLUs to complex Transformer blocks with residual connections and hard-routing (see Figure 2).
Key Results
- Mathematical Precision: On VGG-16, the proposed rules satisfy conservation identities to single-precision round-off error.
- Robustness: $LRP_0$ (Gradient $\times$ Input) was found to be highly sensitive to noise, with a tiny perturbation ($10^{-5}$ std dev) shifting the map by 1.5%. In contrast, the EOT-based rules were 15x-25x more stable.
- Superior Faithfulness: On the modern ConvNeXt-Tiny architecture, the Hebbian-weighted $LRPN_{\alpha,\beta}$ rule achieved the highest faithfulness (measured by ABPC) among all tested rules.
- Region Specificity: Using Schrödinger bridges, the authors showed that in VGG-16, the animal's head receives roughly 2.5 times more relevance per pixel than the body, and that these two regions are processed through largely distinct hidden channels.
Limitations and Future Work
- Computational Overhead: Sinkhorn Relevance Propagation (SRP) requires iterative sweeps, making it significantly slower (22s per image on an NVIDIA T1000) compared to standard LRP (0.16s).
- Scaling to LLMs: While the theory is designed to handle the discontinuities found in MoE-based Large Language Models, the empirical evaluation in this paper is focused on image classifiers. Applying these tools to token-based attribution in LLMs is a clear next step.
- Soft Constraints: The authors suggest that future work could relax the strict conservation constraint into a "soft" penalty using unbalanced optimal transport, which might be more appropriate for certain modeling goals.
Summary updated