In classical financial theory, the problem of finding an optimal trading strategy is almost always framed as a convex optimization problem. This is not because the world is inherently convex, but because convexity makes the math work: it guarantees that a local optimum is a global one and allows researchers to use powerful tools like duality theory and Karush-Kuhn-Tucker (KKT) conditions.
However, real-world markets are defined by "jagged" restrictions. Minimum investment thresholds, discrete lot sizes, cardinality limits (capping the number of stocks held), and binary "buy or sell" decisions are all fundamentally non-convex. Recent research across stochastic calculus, deep learning, and control theory is moving beyond these mathematical simplifications to solve the utility maximization problem in its most realistic, constrained form.
The Failure of Convexity in the Real World
Standard models assume that the set of allowed trading strategies is convex—meaning if you can invest \$100 or \$200, you can also invest any amount in between. As noted by Ying Hu and Peter Imkeller in Utility maximization in incomplete markets, this is often unrealistic. Minimum buy-in requirements or discrete investment tiers create "holes" in the feasible set, rendering traditional convex duality methods inapplicable.
These non-convexities appear in several forms:
- Discrete Requirements: Minimum trade sizes or fixed transaction costs (Oktay Gunluk).
- Combinatorial Limits: Restricting a portfolio to exactly $N$ assets to reduce management overhead (Chao Zhang).
- Lower-Bound Constraints: While upper bounds (caps) are easy for convex solvers, lower bounds often break the required mathematical structure (Felix Bünning).
Bypassing Duality with BSDEs
When convexity breaks, the standard "dual" approach to optimization fails. To solve this, researchers like Ying Hu and Patrick Cheridito have turned to Backward Stochastic Differential Equations (BSDEs).
In OPTIMAL CONSUMPTION AND INVESTMENT IN INCOMPLETE MARKETS WITH GENERAL CONSTRAINTS, they demonstrate that by using measurable selection techniques, one can prove the existence of optimal strategies even when the constraint sets are merely "closed" rather than convex. This shift allows for the modeling of "messy" market restrictions—such as disjoint investment intervals—without losing the ability to identify a predictable optimal control.
Approximation and Differentiable Layers
While stochastic calculus provides the theory for existence, other researchers focus on computational tractability. The challenge is that non-convex constraints often make problems NP-hard or incompatible with the gradient-based methods used in modern AI.
- Successive Convex Approximation (SCA): Daniel P. Palomar and colleagues use SCA to handle non-convexity in risk parity and high-order portfolios, iteratively linearizing the complex constraints to find high-quality solutions (riskParityPortfolio).
- Differentiable Neural Layers: In A Universal End-to-End Approach to Portfolio Optimization via Deep Learning, Zihao Zhang and others introduce specialized neural network layers that can enforce discrete leverage or position limits while remaining end-to-end trainable.
- Convex Relaxation: Stephen P. Boyd and Nicholas Moehle utilize "borrowed curvature" to relax non-convex tax liabilities into a convex form, enabling massive speedups for individualized account management (Tax-Aware Portfolio Construction via Convex Optimization).
Intuition: From Smooth Landscapes to Jagged Terrain
If traditional optimization is like finding the lowest point in a smooth, bowl-shaped valley, non-convex optimization is like finding the lowest point in a rocky terrain filled with fences and forbidden zones.
The research direction identified here suggests that we no longer need to "flatten" the terrain to solve the problem. By using BSDEs to handle the randomness and advanced approximation techniques to handle the "fences," practitioners can now design portfolios that respect the actual, jagged constraints of the financial world.