Balancing Interpretability and Performance in Reinforcement Learning: An Adaptive Spectral Based Linear Approach

Yao Wang, Qianxin Yi, Shao-Bo Lin, Jun Fan, Yao Wang 0003arXiv 2025 · DOI · arXiv

Abstract

Reinforcement Learning (RL) in high-stakes domains like healthcare and finance faces a critical trade-off: the high performance of "black-box" neural networks versus the transparency of linear models. This paper presents a novel framework, Adaptive Spectral Based Linear Q-Learning (SB-LinQL ada), which bridges this gap. By leveraging spectral filter functions and an automated, bias-variance-guided parameter selection strategy, the proposed method achieves performance comparable to Deep RL while maintaining the inherent interpretability of linear models. Empirical results on large-scale recommendation and advertising datasets demonstrate that the approach is both computationally efficient and highly transparent.

Introduction: The Interpretability-Performance Trade-off

In managerial environments—such as personalized medicine or dynamic pricing—decision-makers face a frustrating trade-off. Complex models like Deep RL offer top-tier performance but operate as "black boxes," making it impossible to explain why a specific action was taken. Conversely, traditional linear models are easy to interpret but often fail to capture complex patterns, leading to suboptimal decisions.

While "post-hoc" explanation tools exist, they are often unreliable or misleading. This research addresses the challenge of creating an inherently interpretable model that doesn't sacrifice performance, specifically focusing on "Batch RL," where agents must learn from fixed historical datasets rather than through active trial and error.

Figure 1: The trade-off between performance and interpretability in RL models and the proposed adaptive spectral approach.

Methodology: Adaptive Spectral Based Linear Q-Learning

The authors introduce Adaptive Spectral Based Linear Q-Learning (SB-LinQL ada). The core innovation lies in two areas:

  1. Spectral Filtering: Standard ridge regression often suffers from a "saturation phenomenon" where more data doesn't lead to better results. To overcome this, the authors use spectral filter functions (including Gradient Descent and Spectral Cut-off). These allow the model to better handle "noisy" data dimensions and capture complex structures without losing the simplicity of a linear equation ($Q(s,a) = \langle x, \theta \rangle$).
  2. Adaptive Parameter Selection: Tuning regularization parameters typically requires an exhaustive grid search. This paper proposes a strategy guided by the bias-variance trade-off principle. The algorithm automatically selects the optimal regularization level by analyzing the error decomposition, making the model scalable for real-world applications.

Experimental Evaluation

The framework was evaluated on three distinct environments:

  • Synthetic Simulations: A controlled environment used to compare learned feature weights against known "ground truth" parameters.
  • KuaiRand-1K: A large-scale dataset from the Kuaishou video platform containing 11.7 million interactions.
  • Ali_Display_Ad_Click: A real-world dataset from Taobao (Alibaba) for display advertising, focusing on click-through rate (CTR) prediction.

Key Results and Comparative Analysis

The proposed spectral methods (specifically LGD and LCO) matched or nearly matched the performance of Deep Q-Networks (DNN) and Kernel-based methods while offering significant advantages in efficiency and transparency.

Figure 2: Comparison of RL frameworks across interpretability and performance dimensions.

Performance and Efficiency

  • Training Speed: The spectral linear models were orders of magnitude faster than Deep RL. For example, on the Taobao dataset, the proposed method trained in 6-10 seconds, while the DNN took over 635 seconds.
  • Accuracy: On synthetic data, the spectral approach achieved a significantly lower "parameter gap" (error in estimating true weights) compared to standard Least Squares or LASSO-based RL.

Insights into Transparency

  • The "Less is More" Principle: In both the Kuaishou and Taobao case studies, using a small subset of the most influential features (e.g., the top 4 or 5) yielded better cumulative rewards than using all 34 available features. This suggests that over-parameterization can introduce noise that degrades decision quality.
  • Trust through Visualization: By visualizing feature weights over time, managers can verify that the model is making decisions based on logical factors (like user "active degree" or "city tier") rather than statistical artifacts.

Conclusion and Future Work

While the linear approach is powerful, it relies on the assumption that the underlying Q-function can be reasonably approximated by linear combinations of features. If the environment is extremely non-linear, the performance gap between this and Deep RL might widen. Future work will extend this framework to distributed settings, allowing the model to learn across decentralized data sources without sharing raw, sensitive data, further enhancing privacy and scalability.

Summary updated

Abstract

Reinforcement learning (RL) has been widely applied to sequential decision making, where interpretability and performance are both critical for practical adoption. Current approaches typically focus on performance and rely on post hoc explanations to account for interpretability. Different from these approaches, we focus on designing an interpretability-oriented yet performance-enhanced RL approach. Specifically, we propose a spectral based linear RL method that extends the ridge regression-based approach through a spectral filter function. The proposed method clarifies the role of regularization in controlling estimation error and further enables the design of an adaptive regularization parameter selection strategy guided by the bias-variance trade-off principle. Theoretical analysis establishes near-optimal bounds for both parameter estimation and generalization error. Extensive experiments on simulated environments and real-world datasets from Kuaishou and Taobao demonstrate that our method either outperforms or matches existing baselines in decision quality. We also conduct interpretability analyses to illustrate how the learned policies make decisions, thereby enhancing user trust. These results highlight the potential of our approach to bridge the gap between RL theory and practical decision making, providing interpretability, accuracy, and adaptability in management contexts.
BibTeX
@misc{wang2025balancing,
  author = {Yao Wang and Qianxin Yi and Shao-Bo Lin and Jun Fan and Yao Wang},
  title = {{Balancing Interpretability and Performance in Reinforcement Learning: An Adaptive Spectral Based Linear Approach}},
  year = {2025},
  eprint = {2510.03722},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.03722},
}
Figure