Abstract
Reinforcement Learning (RL) in high-stakes domains like healthcare and finance faces a critical trade-off: the high performance of "black-box" neural networks versus the transparency of linear models. This paper presents a novel framework, Adaptive Spectral Based Linear Q-Learning (SB-LinQL ada), which bridges this gap. By leveraging spectral filter functions and an automated, bias-variance-guided parameter selection strategy, the proposed method achieves performance comparable to Deep RL while maintaining the inherent interpretability of linear models. Empirical results on large-scale recommendation and advertising datasets demonstrate that the approach is both computationally efficient and highly transparent.
Introduction: The Interpretability-Performance Trade-off
In managerial environments—such as personalized medicine or dynamic pricing—decision-makers face a frustrating trade-off. Complex models like Deep RL offer top-tier performance but operate as "black boxes," making it impossible to explain why a specific action was taken. Conversely, traditional linear models are easy to interpret but often fail to capture complex patterns, leading to suboptimal decisions.
While "post-hoc" explanation tools exist, they are often unreliable or misleading. This research addresses the challenge of creating an inherently interpretable model that doesn't sacrifice performance, specifically focusing on "Batch RL," where agents must learn from fixed historical datasets rather than through active trial and error.
Methodology: Adaptive Spectral Based Linear Q-Learning
The authors introduce Adaptive Spectral Based Linear Q-Learning (SB-LinQL ada). The core innovation lies in two areas:
- Spectral Filtering: Standard ridge regression often suffers from a "saturation phenomenon" where more data doesn't lead to better results. To overcome this, the authors use spectral filter functions (including Gradient Descent and Spectral Cut-off). These allow the model to better handle "noisy" data dimensions and capture complex structures without losing the simplicity of a linear equation ($Q(s,a) = \langle x, \theta \rangle$).
- Adaptive Parameter Selection: Tuning regularization parameters typically requires an exhaustive grid search. This paper proposes a strategy guided by the bias-variance trade-off principle. The algorithm automatically selects the optimal regularization level by analyzing the error decomposition, making the model scalable for real-world applications.
Experimental Evaluation
The framework was evaluated on three distinct environments:
- Synthetic Simulations: A controlled environment used to compare learned feature weights against known "ground truth" parameters.
- KuaiRand-1K: A large-scale dataset from the Kuaishou video platform containing 11.7 million interactions.
- Ali_Display_Ad_Click: A real-world dataset from Taobao (Alibaba) for display advertising, focusing on click-through rate (CTR) prediction.
Key Results and Comparative Analysis
The proposed spectral methods (specifically LGD and LCO) matched or nearly matched the performance of Deep Q-Networks (DNN) and Kernel-based methods while offering significant advantages in efficiency and transparency.
Performance and Efficiency
- Training Speed: The spectral linear models were orders of magnitude faster than Deep RL. For example, on the Taobao dataset, the proposed method trained in 6-10 seconds, while the DNN took over 635 seconds.
- Accuracy: On synthetic data, the spectral approach achieved a significantly lower "parameter gap" (error in estimating true weights) compared to standard Least Squares or LASSO-based RL.
Insights into Transparency
- The "Less is More" Principle: In both the Kuaishou and Taobao case studies, using a small subset of the most influential features (e.g., the top 4 or 5) yielded better cumulative rewards than using all 34 available features. This suggests that over-parameterization can introduce noise that degrades decision quality.
- Trust through Visualization: By visualizing feature weights over time, managers can verify that the model is making decisions based on logical factors (like user "active degree" or "city tier") rather than statistical artifacts.
Conclusion and Future Work
While the linear approach is powerful, it relies on the assumption that the underlying Q-function can be reasonably approximated by linear combinations of features. If the environment is extremely non-linear, the performance gap between this and Deep RL might widen. Future work will extend this framework to distributed settings, allowing the model to learn across decentralized data sources without sharing raw, sensitive data, further enhancing privacy and scalability.
Summary updated