Safety-Aware Optimal Control of Stochastic Systems Using Conditional Value-at-Risk

Samantha Samuelson, Insoon Yang

Abstract

This paper introduces a unified framework for controlling stochastic systems that balances cost-minimization with safety by using Conditional Value-at-Risk (CVaR). By reformulating complex risk constraints into a tractable dynamic programming problem, the authors provide a way to design controllers that don't just avoid failure, but specifically limit the severity of potential safety violations.

Introduction

In safety-critical systems—such as autonomous drones or power grids—uncertainty is a fundamental challenge. Traditional "safe" control often relies on probabilistic safe sets, ensuring the system remains safe with a high probability (e.g., 95%). However, these methods possess a significant blind spot: they do not distinguish between a "near miss" and a "catastrophic crash." If a failure occurs, the controller remains indifferent to its severity.

Furthermore, existing methods often decouple safety verification from optimal control, leading to suboptimal performance. This research proposes a "safety-aware" controller that minimizes operational costs while strictly bounding the risk of unsafe states in a computationally feasible manner for real-time applications.

Methodology

The authors propose a risk-based approach utilizing two primary tools:

  1. Set Distance: "Unsafety" is measured as the distance between the system state and a predefined safe set $A$. Loss is zero within $A$ and grows proportionally with the distance outside it.
  2. Conditional Value-at-Risk (CVaR): Rather than focusing on the probability of failure (Value-at-Risk), CVaR evaluates the expected value of the loss in the worst-case tail of the distribution (e.g., the worst 5% of outcomes).

Reformulation for Tractability

Standard CVaR constraints lead to "time-inconsistent" problems that are incompatible with standard dynamic programming. The authors prove that the CVaR constraint can be reformulated into a tractable expectation constraint within a Bellman equation. For systems with finitely supported noise, the problem simplifies into a deterministic convex optimization problem, allowing the use of standard convex solvers.

Numerical Evaluation

The approach was evaluated using a Stochastic Inventory Control model. The system manages stock levels under uncertain demand (modeled as a Gaussian distribution $N(20, 6)$). The objective is to minimize holding and lost-sales costs while maintaining inventory within a "safe" range $[0, 100]$. The disturbance distribution was represented using 40 demand samples.

Results and Discussion

The numerical results highlight several key insights:

  • Severity Matters: By applying CVaR to the distance from the safe set, the controller is incentivized to remain near the center of the safe region. If it must venture near the edge, it does so in a way that minimizes the "depth" of a potential violation.
  • The Risk-Performance Tradeoff: There is a clear "price of safety." As the risk tolerance ($\delta$) decreases (implying tighter safety), the expected minimal cost increases significantly.
The minimal cost increases as the risk tolerance $\delta$ is reduced, illustrating the fundamental tradeoff between safety and performance.
  • Safe Set Dynamics: The "risk-constrained safe set" emerges naturally from the value function calculation. A state is considered safe if the optimization has a solution ($v_0(x) < \infty$). As risk constraints become stricter, the feasible region of initial states shrinks.
The value function $v_0$ as a function of the initial state $x_0$ for different risk levels $\delta$. Lower $\delta$ values result in higher costs and a smaller range of feasible initial states.

Conclusion

The proposed CVaR-based framework successfully integrates safety constraints into optimal control for stochastic systems. While the reformulation is efficient, the underlying dynamic programming still faces the "curse of dimensionality," limiting its application to low-dimensional systems. Future research will explore occupation measure-based methods for higher-dimensional systems and distributionally robust control to account for uncertainties in noise distributions.

Summary updated

Abstract

In this paper, we consider a multi-objective control problem for stochastic systems that seeks to minimize a cost of interest while ensuring safety. We introduce a novel measure of safety risk using the conditional value-at-risk and a set distance to formulate a safety risk-constrained optimal control problem. Our reformulation method using an extremal representation of the safety risk measure provides a computationally tractable dynamic programming solution. A useful byproduct of the proposed solution is the notion of a risk-constrained safe set, which is a new stochastic safety verification tool. We also establish useful connections between the risk-constrained safe sets and the popular probabilistic safe sets. The tradeoff between the risk tolerance and the mean performance of our controller is examined through an inventory control problem.
BibTeX
@inproceedings{samuelson2018safety,
  author = {Samantha Samuelson and Insoon Yang},
  title = {{Safety-Aware Optimal Control of Stochastic Systems Using Conditional Value-at-Risk}},
  booktitle = {ACC},
  year = {2018},
  url = {https://openreview.net/forum?id=__sJrQ2qmu},
}
Figure