Bridging Interpretability and Optimization: Provably Attribution-Weighted Actor-Critic in Reproducing Kernel Hilbert Spaces

Na Li, Hangguan Shan, Wei Ni, Wenjie Zhang, Xinyu Li

Abstract

This paper introduces RSA2C, a novel reinforcement learning (RL) framework that utilizes game-theoretic feature attributions (SHAP values) to prioritize critical state dimensions during training. By embedding these attributions directly into a kernelized Actor-Critic architecture within Reproducing Kernel Hilbert Spaces (RKHS), the model achieves enhanced performance and stability. Unlike traditional post-hoc explainability methods, RSA2C uses explanations to guide the optimization process itself, providing a mathematically grounded approach to interpretable and efficient RL.

Introduction

Standard Actor-Critic (AC) algorithms often function as "black boxes," where the influence of specific state features on agent decisions remains opaque. Furthermore, most RL training treats every input feature as equally important, which can lead to slow convergence and extreme sensitivity to noise in irrelevant state dimensions. RSA2C (RKHS-SHAP-based Advanced Actor-Critic) addresses these issues by using feature attributions not just for human interpretation, but as an active component of the learning signal, effectively using explanations as an optimizer guide.

Methodology

The RSA2C framework is built upon three core technical pillars designed to bridge the gap between interpretability and performance:

  1. RKHS-Based Architecture: Instead of deep neural networks, the Actor and Critics are instantiated in Reproducing Kernel Hilbert Spaces (RKHS). This allows for non-parametric function approximation with explicit control over the model's geometry and capacity.
  2. Analytical SHAP via Kernels: Calculating Shapley values (SHAP) is typically computationally expensive. By utilizing Kernel Mean Embeddings (KME) and Conditional Mean Embeddings (CME), the authors derive closed-form analytical solutions for these attributions, making them efficient enough for real-time online training.
  3. Attribution-Aware Learning: The computed SHAP values are transformed into Mahalanobis-gated weights. These weights modulate the gradients of the Actor and the targets of the Advantage Critic. Essentially, the agent "pays more attention" to state features that the Value Critic identifies as most influential to the expected reward.
Architecture of the RSA2C framework, illustrating the integration of SHAP-based attribution into the Actor-Critic loop.

Experimental Evaluation

The algorithm was evaluated on several standard continuous-control environments from the Gymnasium library, including Pendulum-v1, BipedalWalker-v3, and the high-dimensional Ant-v5 (105D state space). A Discounted LQR task was also used to verify theoretical convergence rates against known optimal solutions.

Key Results

  • Performance and Efficiency: In the Pendulum-v1 task, RSA2C variants improved final average returns by approximately 47.6% to 49.2% compared to standard kernelized AC methods. Furthermore, the kernel-based approach requires significantly fewer FLOPs than deep RL baselines like SAC or PPO.
  • Robustness to Noise: Under noisy conditions, the RSA2C-CME (off-manifold) variant maintained high stability and low performance variance, whereas deep RL baselines saw significant degradation as noise increased.
  • Inherent Interpretability: The model's focus shifts dynamically during training. In the BipedalWalker task, for instance, the model correctly identified leg contact sensors and joint velocities as the primary drivers of success, providing built-in transparency.

Conclusion and Future Work

While RSA2C provides a provably stable and interpretable framework, its linear RKHS formulation faces challenges in capturing the highly nonlinear structures of extremely high-dimensional tasks like Ant-v5, where deep RL eventually achieves higher peak performance. Future research will focus on combining the representational power of deep learning with the provable interpretability and optimization benefits of kernel-based attribution methods to create the next generation of explainable RL agents.

Summary updated

BibTeX
@inproceedings{li2026bridging,
  author = {Na Li and Hangguan Shan and Wei Ni and Wenjie Zhang and Xinyu Li},
  title = {{Bridging Interpretability and Optimization: Provably Attribution-Weighted Actor-Critic in Reproducing Kernel Hilbert Spaces}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=b0Z5xWy23m},
}
Figure