This paper, recently accepted to NeurIPS 2026, introduces DIOS (Distribution-Informed Optimal Stopping), a new framework for training deep learning models to make high-stakes "when to quit" decisions. By focusing on the worst-case outcomes rather than just the average, and by ditching the computationally expensive "critic" networks usually required for risk-sensitive AI, the authors achieve state-of-the-art performance in financial option pricing while significantly slashing training times.
TL;DR
The authors propose DIOS, an efficient algorithm for risk-sensitive deep optimal stopping that optimizes for the Conditional Value at Risk (CVaR). By exploiting the unique structure of stopping problems, DIOS avoids the need for complex state augmentation and auxiliary networks, outperforming existing baselines in both reward quality and training speed.
What Problem Does This Paper Address?
In many real-world scenarios—like deciding when to exercise a stock option, when to stop a medical treatment, or when a self-driving car should take emergency action—the "average" outcome doesn't matter as much as the "worst-case" outcome. This is optimal stopping.
Standard Reinforcement Learning (RL) usually tries to maximize the expected reward. However, in high-stakes environments, we care about risk-sensitivity. A common metric for this is CVaR, which looks at the average of the bottom $X\%$ of outcomes (the "tail").
Applying risk-sensitive RL to optimal stopping is notoriously difficult due to four hurdles:
- State Augmentation: Most methods require adding extra variables to the "state" to keep track of accumulated risk, making the problem harder to solve.
- Gradient Pathology: Standard "score-function" gradients (like REINFORCE) become unstable or vanish when a policy becomes nearly certain (deterministic).
- Blindness to Success: Risk-averse agents often ignore "successful" trajectories during training, wasting valuable data.
- Auxiliary Networks: Many methods require a second "critic" network to estimate risk levels, adding computational overhead.
Methods
The authors introduce a theoretical framework that identifies three structural properties unique to optimal stopping: single-point rewards, action-independent dynamics (the world keeps moving regardless of whether you've stopped), and sum-zero gradients.
Using these, they developed DIOS:
- Pathwise Gradient Estimator: Instead of the unstable score-function approach, they derive a pathwise estimator. This remains stable even when the model is very confident in its decisions.
- Markov Optimality without Augmentation: They prove that for CVaR stopping, you don't actually need to augment the state, keeping the model simple.
- Tail-Level Annealing: To solve "blindness to success," DIOS starts by training on the entire distribution of outcomes and gradually narrows its focus to the risky "tail" as training progresses.
- Batch-Direct Quantile Estimation: Instead of a separate critic network, DIOS estimates the risk threshold (the quantile) directly from each batch of data, making it much faster.
Datasets
The researchers validated DIOS against several baselines (like CVaR-PG, MIX, and RET-CAP) using two primary domains:
- Bermudan Max-Call Options: A classic, high-dimensional financial benchmark. They tested 24 different configurations varying in asset count (up to 200) and volatility.
- House-Selling Problem: A sequential decision task where an agent receives offers and must decide when to accept, considering the costs of waiting. They used both Lognormal and Pareto distributions for the offers.
Key Insights and Ideas
- Simplicity Wins: By proving that state augmentation isn't necessary for Markov optimality in this specific context, the authors show that optimal stopping is "special" compared to general RL.
- Differentiating the Probability, Not the Reward: The pathwise approach allows the model to learn from how the stopping probability changes, rather than just the final outcome. This prevents the "vanishing gradient" problem that plagues other risk-sensitive methods.
- The Power of Annealing: Training a model to care only about the bottom 5% of outcomes from day one is hard because it ignores 95% of the data. Starting at 100% (risk-neutral) and slowly "annealing" down to 5% allows the model to learn the general dynamics before specializing in risk management.
Key Results
- Performance: Across all 24 Bermudan max-call configurations, DIOS achieved the highest mean CVaR.
- Efficiency: DIOS was significantly faster than the competition. It trained 4–5x faster than CVaR-VaR and roughly 2x faster than MIX and RET-CAP.
- Stability: Unlike other methods (like CVaR-PG), DIOS did not suffer from "collapsed" seeds where the model fails to learn anything useful and simply stops immediately.
Limitations and Future Work
- Theoretical Assumptions: The convergence guarantees rely on certain regularity assumptions (like bounded policy derivatives) that might be hard to verify for complex architectures using ReLU activations or BatchNorm.
- Global Optimality: While the paper guarantees finding a stationary point (a local optimum), it doesn't guarantee finding the absolute best global policy.
- Future Directions: The authors suggest extending this work to action-dependent dynamics (where the agent's choice changes the environment) and applying it to more complex medical or industrial control problems where tail-risk is a matter of safety, not just money.
Summary updated