Abstract
The convex analytic method has proved to be a very versatile method for the study of infinite horizon average cost optimal stochastic control problems. In this paper, we revisit the convex analytic method and make three primary contributions: (i) We present an existence result for controlled Markov models that lack weak continuity of the transition kernel but are strongly continuous in the action variable for every fixed state variable. (ii) For average cost stochastic control problems in standard Borel spaces, while existing results establish the optimality of stationary (possibly randomized) policies, few results are available on the optimality of deterministic policies. We review existing results and present further conditions under which an average cost optimal stochastic control problem admits optimal solutions that are deterministic stationary. (iii) We establish conditions under which the performance under stationary deterministic (and also quantized) policies is dense in the set of performance values under randomized stationary policies.
Introduction
In the study of stochastic control—ranging from robotic navigation to financial portfolio management—the primary objective is often to minimize a long-term average cost. Traditionally, researchers have relied on the Bellman equation and dynamic programming to identify optimal policies. However, the convex analytic method offers a powerful alternative by treating the control problem as a linear program over a space of probability measures.
Despite its versatility, the convex analytic method has historically faced two significant hurdles: the requirement for "weak continuity" in system transitions and the theoretical tendency to produce randomized policies. This paper addresses these gaps, providing new existence results and proving that simple, deterministic rules can achieve the same performance as complex, randomized ones.
Overcoming Continuity Gaps
A standard requirement in stochastic control theory is that the transition kernel must be weakly continuous. This implies that small changes in the current state or action result in only small changes in the probability distribution of the next state. In many real-world applications, particularly those involving additive noise with specific densities, this condition is violated.
To address this, the authors utilize the $w-s$ topology. Unlike the standard weak topology, the $w-s$ topology allows for "setwise" convergence in the state variable while maintaining "weak" convergence in the action variable. This refinement allows for the existence of optimal policies even when the system transitions are "jumpy" or lack traditional weak continuity, provided the kernel is strongly continuous in the action variable for every fixed state.
The Randomization Problem and Deterministic Optimality
In theoretical models, the optimal policy is often found to be randomized (e.g., "if the battery is low, there is a 40% chance of charging and a 60% chance of continuing"). From an engineering perspective, such policies are often undesirable due to their complexity and unpredictability. Engineers prefer deterministic policies—fixed rules that dictate a specific action for every state.
Occupation Measures and Extreme Points
The paper transforms the dynamic control problem into a static optimization problem by focusing on occupation measures. These measures represent the long-term proportion of time the system spends in specific state-action pairs. Because the set of all possible occupation measures is convex, the optimal solution must reside at an "extreme point" of this set. The authors demonstrate that these extreme points often correspond to deterministic policies, providing a mathematical justification for the use of non-randomized rules.
Key Theoretical Results
The paper's contributions are formalized through three primary theorems:
- Theorem 2.2 (Existence): This theorem establishes that an optimal policy exists under the relaxed $w-s$ continuity conditions. This significantly expands the class of solvable stochastic control problems to include systems previously considered mathematically intractable.
- Theorem 3.1 (Deterministic Optimality): The authors provide concrete conditions, such as Harris recurrence (the property where a system eventually visits every significant part of the state space), under which the optimal policy is guaranteed to be deterministic.
- Theorem 4.2 (Approximation and Denseness): Perhaps the most practical result, this theorem proves that the performance of deterministic and stationary policies is "dense" within the set of performance values of randomized policies. This means that for any randomized policy, there exists a deterministic (or even a quantized) policy that can approximate its performance with arbitrary precision.
Conclusion and Future Directions
By refining the convex analytic method, this research bridges the gap between abstract control theory and practical engineering requirements. The results show that deterministic policies are not just a convenient simplification but are often theoretically optimal or near-optimal.
While the paper significantly relaxes continuity requirements, achieving deterministic optimality still frequently relies on conditions like Harris recurrence or countable action spaces. Future research may explore even more general conditions to guarantee deterministic optimality in uncountable, non-recurrent systems, further expanding the reach of the convex analytic method in complex, real-world environments.
Summary updated