MQLV: Optimal Policy of Money Management in Retail Banking with Q-Learning

Jérémy Charlier, Gaston Ormazabal, Radu State, Jean Hilger

TL;DR

This paper introduces MQLV, a novel reinforcement learning (RL) approach that uses Q-learning and the Vasicek model to determine optimal money management policies for retail banking clients. MQLV aims to provide personalized financial recommendations, such as credit limits and loan assessments, by learning from aggregated transaction data and estimating future event probabilities like payment defaults.

What Problem Does This Paper Address?

Traditional financial modeling often relies on simplified stochastic processes like Geometric Brownian Motion (GBM), which are limited in their ability to capture the complexities of real-world financial data, particularly in retail banking. Existing RL applications in finance, like QLBS, are often tailored to specific problems like vanilla option pricing and don't extend to broader money management tasks. The paper addresses the need for a more sophisticated and adaptable RL framework that can learn optimal financial policies from diverse transaction data, enabling more personalized and transparent decision-making in retail banking, such as for loan applications and credit card limit setting.

Methods

The core of MQLV is the application of the Q-learning algorithm, a well-established reinforcement learning technique, to a financial context. Unlike previous approaches that might use GBM, MQLV employs the Vasicek model, a mean-reverting stochastic diffusion process, to better simulate the dynamics of retail banking transactions. A key innovation is the introduction of a "digital function," a discrete Heaviside step function, which is used to estimate the probability of future events, such as a client's payment default. This allows MQLV to move beyond simple option pricing and tackle more complex decision-making scenarios. The method is designed to be model-free and off-policy, meaning it doesn't require prior knowledge of the underlying data dynamics and can learn from past experiences.

Datasets

The paper utilizes synthetic datasets generated using the Vasicek model. These datasets are created to mimic anonymized historical retail banking transactions, including credit card and payment data. Three distinct datasets were generated with varying numbers of Monte Carlo paths (20,000, 30,000, and 40,000) to assess the impact of sampling size. Additionally, a separate set of experiments used new Vasicek parameters to test the model's universality. The authors note that real retail banking transaction data cannot be publicly released due to privacy and confidentiality concerns.

Key Insights and Ideas

  • Vasicek Model for Retail Banking: The paper demonstrates that the Vasicek model, a mean-reverting process, can effectively capture the dynamics of retail banking transactions, showing similarities to anonymized historical data. This suggests it's a more suitable model than simpler GBM for this domain.
  • Digital Function for Default Probability: The introduction of a discrete Heaviside step function as a "digital function" is a novel way to estimate future event probabilities, specifically payment defaults, which is crucial for risk assessment in banking.
  • MQLV's Accuracy: MQLV not only learns an optimal policy but also achieves accurate event probability estimations, often outperforming the traditional Black-Scholes-Merton (BSM) closed-form formula, especially in specific configurations.
  • Model-Free and Off-Policy Advantage: The model-free and off-policy nature of MQLV makes it adaptable to real-world banking data without requiring strict assumptions about data dynamics or explicit policy definitions.

Key Results

  • Transaction Similarity: MQLV successfully demonstrated similarities between anonymized historical transactions and Vasicek-generated transactions, with a small RMSE of 0.03.
  • Event Probability Accuracy: MQLV's event probability estimations were found to be close to those derived from the BSM's closed-form formula, with a total RMSE of 1.502 for a specific configuration.
  • Superiority over BSM: In several experimental configurations, MQLV provided more accurate event probabilities than the BSM's closed-form formula, particularly when estimating a 50% probability at a strike of 1. This suggests MQLV's learned policy is more aligned with the true underlying dynamics.
  • Universality: Experiments with varying Vasicek parameters showed that MQLV is capable of learning optimal policies and estimating probabilities accurately, indicating its robustness across different data generation scenarios.

Limitations and Future Work

  • Data Confidentiality: The inability to use real, anonymized retail banking transaction data for public release is a significant limitation, necessitating the use of synthetic data.
  • BSM Formula Underestimation: The paper notes that the BSM's closed-form formula surprisingly underestimated event probabilities in their experiments, which they attribute to limitations in its theoretical framework.
  • Temporal Discretization Sensitivity: Preliminary observations suggest that the Q-learning update might slightly alter real probability values when using very small temporal discretizations (e.g., Δt = 200), potentially due to basis function approximator errors.
  • Future Work: The authors plan to create a fully anonymized dataset for retail banking transactions, evaluate MQLV on data that violates Vasicek assumptions, investigate the impact of temporal discretization, and explore extending the Q-learning update with a deep learning framework.

Summary updated

Abstract

Reinforcement learning has become one of the best approach to train a computer game emulator capable of human level performance. In a reinforcement learning approach, an optimal value function is learned across a set of actions, or decisions, that leads to a set of states giving different rewards, with the objective to maximize the overall reward. A policy assigns to each state-action pairs an expected return. We call an optimal policy a policy for which the value function is optimal. QLBS, Q-Learner in the Black-Scholes(-Merton) Worlds, applies the reinforcement learning concepts, and noticeably, the popular Q-learning algorithm, to the financial stochastic model of Black, Scholes and Merton. It is, however, specifically optimized for the geometric Brownian motion and the vanilla options. Its range of application is, therefore, limited to vanilla option pricing within the financial markets. We propose MQLV, Modified Q-Learner for the Vasicek model, a new reinforcement learning approach that determines the optimal policy of money management based on the aggregated financial transactions of the clients. It unlocks new frontiers to establish personalized credit card limits or bank loan applications, targeting the retail banking industry. MQLV extends the simulation to mean reverting stochastic diffusion processes and it uses a digital function, a Heaviside step function expressed in its discrete form, to estimate the probability of a future event such as a payment default. In our experiments, we first show the similarities between a set of historical financial transactions and Vasicek generated transactions and, then, we underline the potential of MQLV on generated Monte Carlo simulations. Finally, MQLV is the first Q-learning Vasicek-based methodology addressing transparent decision making processes in retail banking.
BibTeX
@inproceedings{charlier2019mqlv,
  author = {Jérémy Charlier and Gaston Ormazabal and Radu State and Jean Hilger},
  title = {{MQLV: Optimal Policy of Money Management in Retail Banking with Q-Learning}},
  booktitle = {MIDAS@PKDD},
  year = {2019},
  url = {https://openreview.net/forum?id=2fpuRlYe3WN},
}
Figure