MQLV: Modified Q-Learning for Vasicek Model

Jérémy Charlier, Gaston Ormazabal, Radu State, Jean Hilger

Abstract

Reinforcement learning (RL) has demonstrated human-level performance in complex environments, yet its application in retail banking remains nascent. While models like QLBS (Q-Learner in the Black-Scholes-Merton World) have successfully applied RL to vanilla option pricing, they are optimized for Geometric Brownian Motion (GBM), which fails to capture the mean-reverting nature of retail financial data. We propose MQLV (Modified Q-Learner for the Vasicek model), a reinforcement learning approach designed to optimize money management and credit limit decisions. By extending RL to mean-reverting stochastic diffusion processes and utilizing a discrete Heaviside step function, MQLV estimates the probability of future events, such as payment defaults. Our experiments demonstrate that MQLV accurately reflects historical financial transactions and outperforms standard closed-form approximations in Monte Carlo simulations, providing a transparent framework for decision-making in retail banking.

Introduction

In the realm of quantitative finance, Reinforcement Learning (RL) has primarily been utilized for asset pricing and portfolio optimization. Most of these applications rely on the assumption that financial assets follow Geometric Brownian Motion (GBM), where prices can theoretically grow without bound. However, retail banking metrics—such as monthly credit card spending, account balances, and loan repayments—exhibit distinct behaviors. These variables are typically mean-reverting, fluctuating around a long-term average rather than trending indefinitely.

To address this gap, we introduce the Modified Q-Learner for the Vasicek model (MQLV). MQLV shifts the focus from the "runaway" dynamics of the stock market to the mean-reverting dynamics of personal finance. This model-free, off-policy approach allows financial institutions to determine optimal credit limits and loan approvals by learning directly from transaction data.

The MQLV Framework

The core innovation of MQLV lies in its adaptation of the Q-learning algorithm to the Vasicek model, a mathematical framework specifically suited for mean-reverting processes.

The Digital Function and Probability Estimation

Unlike traditional RL agents that seek to maximize a monetary reward, MQLV utilizes a Digital Function ($\Pi$)—specifically, a discrete form of the Heaviside step function—as a terminal condition. This allows the agent to estimate the probability of a future event, such as a default, rather than a simple expected value. This transformation turns the optimization problem into a robust risk assessment tool.

Algorithmic Components

  1. Fitted Q-Iteration (FQI): MQLV employs FQI to evaluate the quality (Q-value) of specific actions within the environment. This enables the agent to learn the optimal policy through iterative updates.
  2. Regularization via "Dropout": To prevent the overestimation of Q-values—a common pitfall in RL—the authors implement a regularization term based on a binomial distribution. This functions similarly to "dropout" in deep learning, enhancing the model's generalization.
  3. Backward Induction: The optimal policy is derived by solving the Bellman equation backward from a future maturity date ($T$) to the present ($t=0$), ensuring that every decision is informed by future potential outcomes.

Data and Experimental Design

Due to the sensitive nature of retail banking data and regulations such as GDPR, the study utilizes a combination of anonymized real-world data and synthetic datasets.

  • Anonymized Historical Transactions: A sample of real-world spending data was used to validate the choice of the Vasicek model.
  • Synthetic Vasicek Datasets: To ensure reproducibility, the authors generated three large-scale datasets using Monte Carlo simulations, containing 20,000, 30,000, and 40,000 paths respectively. These datasets simulate various mean-reverting scenarios to test the agent's robustness.

Results and Discussion

The performance of MQLV was evaluated based on its ability to mimic real-world data and its accuracy relative to theoretical benchmarks.

Plausibility and Accuracy

The Vasicek-generated transactions showed a high degree of similarity to real-world spending patterns, achieving a Root Mean Square Error (RMSE) of 0.03. This validates the model's underlying assumption that mean reversion is the appropriate lens for retail banking.

Furthermore, MQLV was compared against the standard Black-Scholes-Merton (BSM) closed-form approximation. In scenarios where the theoretical probability of an event was 50%, MQLV consistently provided estimates closer to the target than the BSM formula. This suggests that the RL-based approach is more capable of capturing the nuances of mean-reverting processes than traditional closed-form models.

Transparency and Flexibility

Because MQLV is "model-free," it does not require prior knowledge of the environment's transition probabilities. This makes it highly flexible for real-world banking environments where the underlying "rules" of consumer behavior are complex and often unknown. Additionally, by providing a benchmark for "optimal" money management, MQLV offers a transparent mechanism for banks to justify credit decisions, reducing bias in the lending process.

Conclusion and Future Work

MQLV represents a significant step toward applying advanced RL techniques to the retail banking sector. By moving beyond GBM and embracing mean reversion, the framework provides a more accurate and transparent tool for risk management.

Future research will focus on addressing approximation errors that occur with very small time steps ($\Delta t$) and transitioning from linear basis functions to Deep Q-Learning to capture non-linear financial behaviors. Ultimately, MQLV paves the way for a new generation of personalized, data-driven financial services.

Summary updated

Abstract

Reinforcement learning has become one of the best approach to train a computer game emulator capable of human level performance. In a reinforcement learning approach, an optimal value function is learned across a set of actions, or decisions, that leads to a set of states giving different rewards, with the objective to maximize the overall reward. A policy assigns to each state-action pairs an expected return. We call an optimal policy a policy for which the value function is optimal. QLBS, Q-Learner in the Black-Scholes(-Merton) Worlds, applies the reinforcement learning concepts, and noticeably, the popular Q-learning algorithm, to the financial stochastic model of Black, Scholes and Merton. It is, however, specifically optimized for the geometric Brownian motion and the vanilla options. Its range of application is, therefore, limited to vanilla option pricing within financial markets. We propose MQLV, Modified Q-Learner for the Vasicek model, a new reinforcement learning approach that determines the optimal policy of money management based on the aggregated financial transactions of the clients. It unlocks new frontiers to establish personalized credit card limits or to fulfill bank loan applications, targeting the retail banking industry. MQLV extends the simulation to mean reverting stochastic diffusion processes and it uses a digital function, a Heaviside step function expressed in its discrete form, to estimate the probability of a future event such as a payment default. In our experiments, we first show the similarities between a set of historical financial transactions and Vasicek generated transactions and, then, we underline the potential of MQLV on generated Monte Carlo simulations. Finally, MQLV is the first Q-learning Vasicek-based methodology addressing transparent decision making processes in retail banking.
BibTeX
@inproceedings{charlier2019mqlv,
  author = {Jérémy Charlier and Gaston Ormazabal and Radu State and Jean Hilger},
  title = {{MQLV: Modified Q-Learning for Vasicek Model}},
  booktitle = {MIDAS@PKDD},
  year = {2019},
  url = {https://openreview.net/forum?id=2fpuRlYe3WN},
}
Figure