Optimal Execution via Multi-Objective Multi-Armed Bandits (Student Abstract)

Francois Buet-Golfouse, Peter HillAAAI 2023 · DOI

TL;DR

This paper tackles the challenge of optimally selling a large stock holding by reformulating it as a multi-objective problem. They propose a novel multi-armed bandit algorithm that balances maximizing profit with minimizing adverse price impact, allowing for user-defined risk appetites.

What Problem Does This Paper Address?

When an investor needs to sell a large quantity of stock, their own trades can significantly impact the market price, leading to a lower-than-expected return. The "optimal execution" problem aims to find a strategy to break down this large trade into smaller, sequential trades over time to minimize this price impact while still achieving a good overall profit. Traditional approaches often focus on a single objective (e.g., maximizing profit). This paper addresses the need to simultaneously consider multiple objectives: maximizing profit and minimizing the negative price movements caused by the trades themselves. They also want to incorporate the user's specific risk tolerance into this decision-making process.

Methods

The authors frame the optimal execution problem as a multi-objective multi-armed bandit (MAB) problem. In a MAB setting, an agent learns to choose the best action from a set of options by observing noisy rewards. Here, the "actions" are the amounts of stock to trade at each time step. The "objectives" are maximizing the total profit and minimizing the total cost (which includes penalties for adverse price movements and holding inventory).

They introduce a novel algorithm called Multi-Objective Contextual Gaussian Process Upper Confidence Bound (CGP-UCB-MO). This algorithm leverages Gaussian Processes (GPs) to model the uncertain reward function. GPs are flexible non-parametric models that can learn from data and provide uncertainty estimates. The "contextual" aspect means the algorithm can use additional information (like current market conditions) to make better decisions. The "multi-objective" part allows it to handle multiple reward signals simultaneously.

A key innovation is the use of temporal credit assignment, where rewards and costs are evaluated at each time step rather than just at the end of the trading period. This allows for more immediate feedback and learning. They also introduce a "preference vector" that allows users to specify their desired trade-off between the objectives, effectively encoding their risk appetite.

Datasets

Not discussed in the paper. The experimentation section describes simulated trading scenarios.

Key Insights and Ideas

  • Multi-objective framing is crucial: Treating optimal execution as a multi-objective problem, rather than a single-objective one, is essential for capturing the complex trade-offs involved.
  • Risk appetite matters: Allowing users to define their preferences (risk appetite) through a preference vector is a powerful way to personalize the execution strategy.
  • Temporal credit assignment improves learning: Evaluating rewards and costs at each step, rather than just at the end, leads to more effective learning in sequential decision-making problems like trading.
  • MABs are well-suited for online trading: The online learning nature of MABs makes them a natural fit for dynamic trading environments where decisions must be made in real-time.

Key Results

The paper presents experimental results comparing their proposed multi-step approach (using CGP-UCB-MO) with a single-step approach.

  • Improved performance: The multi-step approach generally achieves better rewards, especially over longer periods.
  • Trade-off between speed and performance: While the multi-step approach yields better results, it is noted to be slower than the single-step approach.
  • Preference vector effectiveness: Table 2 shows that the mean reward difference between the multi-step and single-step approaches varies with different preference vectors, indicating that the algorithm adapts to user preferences. For example, with a preference of [0.75, 0.25], the multi-step approach shows a positive mean reward difference, suggesting it's superior.

Limitations and Future Work

  • Computational Cost: The multi-step approach, while more effective, is slower. Future work could focus on optimizing its computational efficiency.
  • Observability: The paper assumes that the reward signal is always observable. In real-world trading, this might not always be the case, and future work could explore partial observability.
  • Model Dynamics: The paper uses a simplified model for price dynamics. More complex and realistic market models could be investigated.
  • Real-world data: The experiments are based on simulated data. Applying and evaluating the algorithm on real-world trading data would be a crucial next step.

Summary updated

BibTeX
@inproceedings{buetgolfouse2023optimal,
  author = {Francois Buet-Golfouse and Peter Hill},
  title = {{Optimal Execution via Multi-Objective Multi-Armed Bandits (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26945},
}
Figure