Abstract
This paper studies the optimization of observation channels (stochastic kernels) in partially observed stochastic control problems. In particular, existence, continuity, and convexity properties are investigated. Continuity properties of the optimal cost in channels are explored under total variation, setwise convergence, and weak convergence. Sufficient conditions for sequential compactness under total variation and setwise convergence are presented. It is shown that the optimization is concave in observation channels. This implies that the optimization problem is non-convex in quantization/coding policies for a class of networked control problems. Furthermore, the paper explains why a class of decentralized control problems, under the non-classical information structure, is non-convex when signaling is present.
Introduction
In traditional stochastic control, the observation channel—the mechanism through which a controller receives information about the system state—is typically treated as a fixed constraint. Research focuses primarily on finding the optimal control policy for that specific sensor. However, in modern networked systems, robotics, and communication-constrained environments, the sensor itself is often a design variable.
This paper addresses the "dual problem": if we can design or choose our observation channel, which one is optimal? By treating observation channels as stochastic kernels, the authors establish a rigorous mathematical framework to evaluate how variations in sensor noise, quantization, and uncertainty impact the minimum achievable control cost.
Problem Formulation: Continuity and Existence
The authors focus on two fundamental mathematical challenges essential for robust engineering design:
- Continuity (P1): Does a small change in the sensor model lead to a small change in the optimal cost? This is critical for consistency; if a sensor model is estimated from data, we must ensure that the resulting controller performs predictably as the estimate improves.
- Existence (P2): Within a given set of possible sensors (e.g., all possible 3-bit quantizers), is an optimal sensor guaranteed to exist? Without this guarantee, optimization algorithms may fail to converge or seek a "best" sensor that does not mathematically exist within the feasible set.
Convergence of Observation Channels
The paper analyzes observation channels as stochastic kernels—mappings from a state space to a probability distribution of observations. To study their optimization, the authors evaluate three distinct topologies of convergence:
- Weak Convergence: The most lenient form, where only the "average" behavior of the kernels converges.
- Setwise Convergence: A middle ground where the probability of any specific event converges.
- Total Variation (TV) Convergence: The strictest form, requiring the entire probability distribution to converge uniformly across the state space.
The Gold Standard: Total Variation
A significant insight of the paper is that Weak Convergence is insufficient for ensuring the continuity of the optimal cost. The authors provide counterexamples demonstrating that a sequence of sensors can converge weakly while the optimal cost undergoes a massive, discontinuous jump.
In contrast, the authors prove that Total Variation (TV) is the necessary standard for stability. Under TV convergence, the optimal cost is guaranteed to be continuous (Theorem 3.4). This finding suggests that for high-stakes control applications, sensor design must be robust under the strictest distributional metrics.
Convexity and the Complexity of Signaling
A major contribution of the paper is the analysis of the optimization landscape. The authors demonstrate that while the optimization problem is concave with respect to the observation channels (stochastic kernels), it is often non-convex when viewed through the lens of specific quantization or coding policies.
This distinction explains a long-standing difficulty in decentralized control: why problems involving "signaling" (where an agent's actions are used to communicate information to another agent) are so computationally hard. Under non-classical information structures, the mapping from policies to information channels is non-linear, breaking the convexity that usually makes optimization tractable.
Key Results and Quantization
- Existence of Optimal Quantizers: The paper shows that deterministic quantizers—simple rules that bin continuous data into discrete categories—act as a "basis" for the broader space of randomized channels. Theorem 5.9 proves that for systems with "smooth" (absolutely continuous) state distributions, an optimal $M$-cell quantizer is guaranteed to exist.
- Compactness Conditions: Using tools like the Arzelà-Ascoli and Krein-Milman theorems, the authors provide conditions for the sequential compactness of channel spaces, ensuring that "best-case" and "worst-case" sensors can be identified (Theorem 4.5).
- Multi-stage Systems: The research extends these findings to multi-stage problems. While the "dual effect of control" (where actions influence future information) complicates the analysis, the authors show that continuity holds if sensors converge uniformly across all possible system states.
Conclusion and Future Directions
This work provides the theoretical foundation for sensor-aware control design. By establishing TV convergence as the benchmark for continuity and identifying the inherent non-convexity in signaling policies, the authors clarify why certain decentralized problems remain elusive.
Future research may look toward extending these results to systems with unbounded cost functions (such as the quadratic costs in LQG control) and developing more general theories for the multi-stage case, where the interaction between control and information is most dynamic.
Summary updated