Policy iteration using Q-functions: Linear dynamics with multiplicative noise

Peter Coppens, Panagiotis PatrinosarXiv 2022 · DOI · arXiv

This paper, titled "Policy iteration using Q-functions: Linear dynamics with multiplicative noise," presents a significant step forward in making reinforcement learning (RL) more robust for real-world physical systems. Authors Peter Coppens and Panagiotis Patrinos bridge the gap between classical control theory and modern data-driven learning.

TL;DR

The authors develop a model-free, data-driven algorithm to solve the Linear Quadratic Regulator (LQR) problem for systems where noise scales with the state and input (multiplicative noise). By using Q-functions and "instrumental variables," they create a policy iteration scheme that matches the performance of model-based methods without needing to explicitly identify the system's underlying physics.

What Problem Does This Paper Address?

In classical control (like standard LQR), we often assume "additive noise"—a random nudge that hits the system regardless of what it’s doing. However, in many real-world scenarios (aerospace, biology, communication channels), noise is multiplicative: the harder you push the engine or the further you move, the more uncertainty you encounter.

Multiplicative noise breaks several standard "shortcuts" in control theory:

  1. The Separation Principle fails: You can no longer design the estimator and the controller independently.
  2. Bias in Data: If you try to learn the system's value from data using standard least-squares regression, your estimates will be biased (inconsistent) because the noise is mathematically "tangled" with your data points.
  3. Model-Free Gap: Most existing solutions either require a known model or use Policy Gradient methods, which can be notoriously slow and sample-inefficient.

Methods

The authors propose a Policy Iteration (PI) scheme. PI involves two steps: Policy Evaluation (figuring out how good the current strategy is) and Policy Improvement (finding a better strategy based on that evaluation).

The key technical innovations are:

  • Q-functions for Model-Free Control: By learning a Q-function (which maps state-action pairs to costs) instead of just a Value function, the algorithm can improve the policy without ever needing to know the $A$ and $B$ matrices of the system dynamics.
  • Instrumental Variables (IV): To solve the bias problem in the evaluation step, the authors use the "Instrumental Variables" technique. They use past state information as an "instrument" that is correlated with the system's behavior but uncorrelated with the current noise, allowing for an unbiased, consistent estimate of the Q-function.
  • Recursive Implementation: They provide a recursive version of the algorithm (similar to a Kalman filter update), making it computationally efficient for online learning.

Datasets

The paper evaluates the algorithm using a numerical simulation of a satellite pitch control problem.

  • The system is discretized and subjected to state- and input-multiplicative noise.
  • The performance is benchmarked against two other approaches: System Identification (SI) (a model-based approach) and Natural Policy Gradient (PG).

Key Insights and Ideas

  • The "Aha" Moment on Bias: The authors realized that the difficulty in data-driven control for multiplicative noise isn't just the complexity of the math, but a fundamental statistical bias. Using Instrumental Variables (a tool often found in econometrics) is the "secret sauce" that makes model-free policy evaluation work here.
  • Parity with Model-Based Methods: Surprisingly, this model-free approach achieves the same "sample complexity" as model-based methods. Both show a suboptimality that decreases at a rate of $1/N$ (where $N$ is the number of samples).
  • Policy Gradient Limitations: The experiments show that while Policy Gradient is easier to implement, it often "stagnates" or requires much more careful tuning and specific data generation compared to the proposed PI scheme.

Key Results

  • Convergence: The algorithm successfully converges to the optimal controller in both "off-policy" (using a fixed data set) and "on-policy" (updating data as the policy improves) settings.
  • Efficiency: The proposed PI scheme performs nearly identically to the model-based System Identification approach in terms of accuracy and sample efficiency.
  • Robustness: The inclusion of multiplicative noise in the design process naturally induces robustness against parametric uncertainty in the system.

Limitations and Future Work

  • Early-Stage Instability: Like many policy iteration schemes, if the algorithm is given too little data in the very first iterations, it can produce an "unstable" controller that causes the system to diverge.
  • Tuning Requirements: There is a confidence parameter ($\beta_0$) that requires some manual tuning. If set incorrectly, the algorithm might trust a bad initial guess too much, leading to failure.
  • Future Directions: The authors suggest investigating "adaptive iteration lengths"—starting with short bursts of data and increasing them as the policy stabilizes—to prevent early divergence. They also aim to further explore the theoretical connection between their method and the Extended Kalman Filter (EKF).

Summary updated

Abstract

This paper presents a novel model-free and fully data-driven policy iteration scheme for quadratic regulation of linear dynamics with state- and input-multiplicative noise. The implementation is similar to the least-squares temporal difference scheme for Markov decision processes, estimating Q-functions by solving a least-squares problem with instrumental variables. The scheme is compared with a model-based system identification scheme and natural policy gradient through numerical experiments.
BibTeX
@misc{coppens2022policy,
  author = {Peter Coppens and Panagiotis Patrinos},
  title = {{Policy iteration using Q-functions: Linear dynamics with multiplicative noise}},
  year = {2022},
  eprint = {2212.01192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.01192},
}
Figure