Abstract
Reinforcement learning (RL) has demonstrated human-level performance in complex environments, yet its application in retail banking remains nascent. While models like QLBS (Q-Learner in the Black-Scholes-Merton World) have successfully applied RL to vanilla option pricing, they are optimized for Geometric Brownian Motion (GBM), which fails to capture the mean-reverting nature of retail financial data. We propose MQLV (Modified Q-Learner for the Vasicek model), a reinforcement learning approach designed to optimize money management and credit limit decisions. By extending RL to mean-reverting stochastic diffusion processes and utilizing a discrete Heaviside step function, MQLV estimates the probability of future events, such as payment defaults. Our experiments demonstrate that MQLV accurately reflects historical financial transactions and outperforms standard closed-form approximations in Monte Carlo simulations, providing a transparent framework for decision-making in retail banking.
Introduction
In the realm of quantitative finance, Reinforcement Learning (RL) has primarily been utilized for asset pricing and portfolio optimization. Most of these applications rely on the assumption that financial assets follow Geometric Brownian Motion (GBM), where prices can theoretically grow without bound. However, retail banking metrics—such as monthly credit card spending, account balances, and loan repayments—exhibit distinct behaviors. These variables are typically mean-reverting, fluctuating around a long-term average rather than trending indefinitely.
To address this gap, we introduce the Modified Q-Learner for the Vasicek model (MQLV). MQLV shifts the focus from the "runaway" dynamics of the stock market to the mean-reverting dynamics of personal finance. This model-free, off-policy approach allows financial institutions to determine optimal credit limits and loan approvals by learning directly from transaction data.
The MQLV Framework
The core innovation of MQLV lies in its adaptation of the Q-learning algorithm to the Vasicek model, a mathematical framework specifically suited for mean-reverting processes.
The Digital Function and Probability Estimation
Unlike traditional RL agents that seek to maximize a monetary reward, MQLV utilizes a Digital Function ($\Pi$)—specifically, a discrete form of the Heaviside step function—as a terminal condition. This allows the agent to estimate the probability of a future event, such as a default, rather than a simple expected value. This transformation turns the optimization problem into a robust risk assessment tool.
Algorithmic Components
- Fitted Q-Iteration (FQI): MQLV employs FQI to evaluate the quality (Q-value) of specific actions within the environment. This enables the agent to learn the optimal policy through iterative updates.
- Regularization via "Dropout": To prevent the overestimation of Q-values—a common pitfall in RL—the authors implement a regularization term based on a binomial distribution. This functions similarly to "dropout" in deep learning, enhancing the model's generalization.
- Backward Induction: The optimal policy is derived by solving the Bellman equation backward from a future maturity date ($T$) to the present ($t=0$), ensuring that every decision is informed by future potential outcomes.
Data and Experimental Design
Due to the sensitive nature of retail banking data and regulations such as GDPR, the study utilizes a combination of anonymized real-world data and synthetic datasets.
- Anonymized Historical Transactions: A sample of real-world spending data was used to validate the choice of the Vasicek model.
- Synthetic Vasicek Datasets: To ensure reproducibility, the authors generated three large-scale datasets using Monte Carlo simulations, containing 20,000, 30,000, and 40,000 paths respectively. These datasets simulate various mean-reverting scenarios to test the agent's robustness.
Results and Discussion
The performance of MQLV was evaluated based on its ability to mimic real-world data and its accuracy relative to theoretical benchmarks.
Plausibility and Accuracy
The Vasicek-generated transactions showed a high degree of similarity to real-world spending patterns, achieving a Root Mean Square Error (RMSE) of 0.03. This validates the model's underlying assumption that mean reversion is the appropriate lens for retail banking.
Furthermore, MQLV was compared against the standard Black-Scholes-Merton (BSM) closed-form approximation. In scenarios where the theoretical probability of an event was 50%, MQLV consistently provided estimates closer to the target than the BSM formula. This suggests that the RL-based approach is more capable of capturing the nuances of mean-reverting processes than traditional closed-form models.
Transparency and Flexibility
Because MQLV is "model-free," it does not require prior knowledge of the environment's transition probabilities. This makes it highly flexible for real-world banking environments where the underlying "rules" of consumer behavior are complex and often unknown. Additionally, by providing a benchmark for "optimal" money management, MQLV offers a transparent mechanism for banks to justify credit decisions, reducing bias in the lending process.
Conclusion and Future Work
MQLV represents a significant step toward applying advanced RL techniques to the retail banking sector. By moving beyond GBM and embracing mean reversion, the framework provides a more accurate and transparent tool for risk management.
Future research will focus on addressing approximation errors that occur with very small time steps ($\Delta t$) and transitioning from linear basis functions to Deep Q-Learning to capture non-linear financial behaviors. Ultimately, MQLV paves the way for a new generation of personalized, data-driven financial services.
Summary updated