Logic-Q: Improving Deep Reinforcement Learning-based Quantitative Trading via Program Sketch-based Tuning

Zhiming Li, Junzhe Jiang, Yushi Cao, Aixin Cui, Bozhi Wu, Bo Li, Yang Liu, Danny Dongning Sun

TL;DR

Logic-Q is a novel neuro-symbolic framework that shields deep reinforcement learning (DRL) trading strategies from catastrophic market crashes. By overlaying a lightweight, human-interpretable "program sketch" onto pre-trained DRL models, it dynamically adjusts trading behavior based on macro market trends without requiring any retraining of the underlying neural network.

What Problem Does This Paper Address?

Deep reinforcement learning (DRL) has become a dominant force in quantitative trading, automating everything from order execution to portfolio allocation. However, these models suffer from a major flaw: they are prone to overfitting to historical noise and struggle to recognize macro-level market shifts. When a market crash hits, even state-of-the-art DRL models often fail to foresee the trend, resulting in massive drawdowns and missed opportunities.

Figure 1: Illustration of the cumulative return curves of two state-of-the-art DRL stock trading strategies during two mar- ket crashes of the US stock market.

While human traders rely on technical indicators to identify macro trends, translating this abstract expertise into a deep learning model is notoriously difficult. Hardcoding rules (e.g., "sell if volatility is high") fails because the exact numerical thresholds change constantly as the market evolves. The authors introduce Logic-Q to bridge this gap, combining the flexibility of neural networks with the robust guardrails of human logic.

Methods

Logic-Q introduces a post-hoc tuning framework based on the program synthesis by sketching paradigm. Instead of retraining the DRL model, Logic-Q leaves the neural network weights completely frozen and applies a lightweight symbolic layer on top.

The framework consists of three main components:

  1. Market Trend-aware Program Sketch: The authors designed a logical template containing conditional statements for five distinct market states: steady descend, steady ascend, rapid descend, rapid ascend, and oscillation. The inputs are three standard technical indicators: volatility, downside risk, and growth rate. Crucially, the exact numerical thresholds are left as "holes" ($\Theta$) to be filled.
  2. Post-hoc Policy Tuning: Based on the market trend identified by the program, it outputs a tuning parameter ($\phi_\tau$) to warp the DRL model's action probabilities:
  3. For Single-Model RL (e.g., Order Execution): It modifies the softmax temperature. If a rapid descent is detected, it smooths the probability distribution to lower the model's confidence and mitigate risk.
  4. For Ensemble RL (e.g., Stock Trading): It acts as a bagging-based weight tensor, dynamically shifting capital toward the sub-policies best suited for the current market trend.
  5. Bayesian Optimization: Because the program sketch is incredibly compact (requiring only 13 parameters for single-model setups and 23 for ensembles), it cannot be trained via backpropagation. Instead, the authors use Bayesian Optimization on a small validation dataset to efficiently find the optimal thresholds.
Figure 2: Overview of the Logic-Q framework.

Datasets

The authors evaluated Logic-Q across two distinct quantitative trading tasks using extensive historical data:

  • Order Execution (OE): Tested on minute-level intraday price-volume transaction data from the China A-shares market (specifically targeting CSI 800 constituent stocks).
  • Stock Trading (ST): Evaluated across three major financial sectors using daily close data from Yahoo Finance:
  • US Market: Dow Jones 30 constituent stocks.
  • Hong Kong Market: Hang Seng China 50 Index constituent stocks.
  • Cryptocurrency Market: Ten high-volume cryptocurrencies paired with USD (including BTC, ETH, and LINK), evaluated under both cash and margin trading scenarios.

Key Insights and Ideas

  • Neuro-Symbolic Synergy: Pure neural approaches excel at capturing micro-structural market patterns, while symbolic logic is far more reliable at enforcing macro-level safety constraints. Logic-Q successfully marries the two.
  • Zero-Retraining Adaptation: Retraining DRL models on financial data is computationally expensive and highly unstable. Logic-Q proves that you can dramatically alter and improve a model's macro behavior simply by warping its output probabilities post-hoc.
  • Extreme Parameter Efficiency: By reducing the macro-trend analyzer to just 13–23 floating-point parameters, the model completely avoids overfitting and can be optimized rapidly using minimal validation data.

Key Results

Logic-Q was pitted against several state-of-the-art DRL baselines, including PPO, OPD, and AlphaMix, yielding substantial performance gains:

  • Order Execution: Logic-Q outperformed the previous best model (OPD), improving the Price Advantage (PA) metric by 23.7% and boosting the Additional Annualized Rate of Return (ARR) by 25.1%.
  • Stock Trading Performance: In the US stock market (cash trading), Logic-Q achieved a Sharpe Ratio (risk-adjusted return) of 1.097, compared to AlphaMix's 0.831.
  • Crash Resilience: During a simulated US market crash, Logic-Q drastically minimized losses. It limited maximum drawdown to -43.3% (cash) and -35.1% (margin), whereas the leading baseline, AlphaMix, suffered drawdowns of -62.6% and -38.7% respectively.

Limitations and Future Work

While Logic-Q demonstrates impressive robustness, its reliance on a fixed, human-designed program sketch means it is bound by the five predefined market states and three chosen technical indicators. If a market behaves in a way that defies these categories, the framework's efficacy might degrade.

The authors note that this is the first paper to use a symbolic model to enhance DRL trading strategies. They suggest that future work could explore expanding the complexity of the program sketches or adapting this framework to other financial domains, such as automated market making.

Summary updated

Abstract

Deep reinforcement learning (DRL) has revolutionized quantitative trading (Q-trading) by achieving decent performance without significant human expert knowledge. Despite its achievements, we observe that the current state-of-the-art DRL models are still ineffective in identifying the market trends, causing them to miss good trading opportunities or suffer from large drawdowns when encountering market crashes. To address this limitation, a natural approach is to incorporate human expert knowledge in identifying market trends. Whereas, such knowledge is abstract and hard to be quantified. In order to effectively leverage abstract human expert knowledge, in this paper, we propose a universal logic-guided deep reinforcement learning framework for Q-trading, called Logic-Q. In particular, Logic-Q adopts the program synthesis by sketching paradigm and introduces a logic-guided model design that leverages a lightweight, plug-and-play market trend-aware program sketch to determine the market trend and correspondingly adjusts the DRL policy in a post-hoc manner. Extensive evaluations of two popular quantitative trading tasks demonstrate that Logic-Q can significantly improve the performance of previous state-of-the-art DRL trading strategies.
BibTeX
@inproceedings{li2025logic,
  author = {Zhiming Li and Junzhe Jiang and Yushi Cao and Aixin Cui and Bozhi Wu and Bo Li and Yang Liu and Danny Dongning Sun},
  title = {{Logic-Q: Improving Deep Reinforcement Learning-based Quantitative Trading via Program Sketch-based Tuning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i17.34045},
}

Loading related content...

Figure