Data-Driven Market-Making via Model-Free Learning

Yueyang Zhong, YeeMan Bergstrom, Amy R. WardIJCAI 2020 · DOI

TL;DR

This paper tackles the challenge of optimizing market-making strategies in electronic trading by using a model-free reinforcement learning approach (Q-learning). They developed a data-driven strategy that learns to place limit orders to maximize profit while managing risk, and successfully demonstrated its superior performance over benchmarks in backtesting, leading to its adoption by a trading firm.

What Problem Does This Paper Address?

Market makers play a crucial role in financial markets by providing liquidity, essentially standing ready to buy or sell assets. This is typically done by placing "limit orders" on an electronic order book (LOB) at specific prices. The core challenge for a market maker is to decide when and at what price to place these orders to maximize their profit, while simultaneously managing the risk of holding too much inventory or being exposed to adverse price movements. Traditional approaches often rely on simplified models of market dynamics, which may not capture the full complexity of real-world trading. This paper aims to address this by developing a data-driven strategy that learns optimal trading policies directly from historical market data, without needing to explicitly model the underlying market dynamics.

Methods

The authors employ a model-free, off-policy reinforcement learning algorithm called Q-learning. This approach is well-suited for problems where the exact dynamics of the environment (in this case, the LOB) are unknown or too complex to model accurately.

Key aspects of their method include:

  • Markov Decision Process (MDP) Formulation: The trading problem is framed as an MDP, where the state represents the LOB's condition, actions are decisions to place or cancel orders, and rewards are based on profit and risk.
  • State Aggregation: To handle the enormous state space of the LOB (due to numerous price levels and order quantities), they use a novel state aggregation technique. This reduces the state space to a manageable size (200 states) by combining several key attributes: bid/ask speed (market volatility), average mid-price change, inventory sign, and cumulative profit and loss.
  • Restricted Action Space: To manage risk, the action space is restricted in certain states, particularly when the firm's inventory is heavily imbalanced.
  • Q-learning Algorithm: The core Q-learning algorithm iteratively updates a Q-table, which stores the expected future reward for taking a specific action in a given aggregated state. This table is then used to derive the optimal trading policy.

Datasets

The primary dataset used in this study is Level II order book data from the Chicago Mercantile Exchange (CME) for a specific futures contract during 2019. This data is characterized by its high granularity (microsecond precision for events like order submission, cancellation, and execution). The data was truncated to a specific trading window (9:00 a.m.

  • 2:30 p.m.) to focus on periods with consistent market dynamics.

Key Insights and Ideas

  • Model-Free Learning is Viable for Market Making: The paper demonstrates that a model-free approach like Q-learning can effectively learn profitable market-making strategies, even without explicit modeling of market dynamics. This is significant because accurately modeling LOB dynamics is notoriously difficult.
  • State Aggregation is Crucial: The success of the Q-learning approach hinges on effective state aggregation. By identifying and combining the most relevant features of the LOB and the firm's position, they significantly reduce the state space, making the learning process tractable.
  • Inventory Management is Key: The learned strategy emphasizes maintaining inventory close to zero, aligning with the idea that market makers should primarily profit from the spread rather than directional bets.
  • Profitability and Risk Management are Intertwined: The objective function and the restricted action space highlight the paper's focus on balancing profit maximization with risk mitigation, particularly through penalties for inventory imbalance and a focus on Sharpe ratio.

Key Results

  • Superior Performance: The proposed Q-learning strategy achieved the highest cumulative PnL in out-of-sample backtesting compared to several benchmarks, including fixed spread strategies and a random strategy.
  • High Sharpe Ratio: The strategy also achieved a Sharpe ratio above 3, which is considered excellent, indicating strong risk-adjusted returns.
  • Production Adoption: The trading firm with whom the authors collaborated found the strategy's performance compelling enough to desire its implementation into production.

Limitations and Future Work

  • Equity Curve Smoothing: The authors acknowledge that the learned strategy can exhibit a phenomenon of several losing days followed by a large gain, leading to a volatile equity curve. This is not ideal from a risk management perspective.
  • Randomized Time Horizon: To address the equity curve smoothing issue, future work will explore developing an algorithmic approach to decide when to close down trading, effectively making the time horizon a random variable. This could lead to a smoother profit realization.

Summary updated

BibTeX
@inproceedings{zhong2020data,
  author = {Yueyang Zhong and YeeMan Bergstrom and Amy R. Ward},
  title = {{Data-Driven Market-Making via Model-Free Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/615},
}
Figure