Abstract
We consider the setting of iterative learning control (ILC), or model-based policy learning in the presence of uncertain, time-varying dynamics. In this setting, we propose a new performance metric, planning regret, which replaces the standard stochastic uncertainty assumptions with worst-case regret. Based on recent advances in non-stochastic control, we design a new iterative algorithm, iGPC, for minimizing planning regret that is more robust to model mismatch and uncertainty. We provide theoretical and empirical evidence that the proposed algorithm outperforms existing methods on several benchmarks.
Introduction
In robotics and control, the "nominal model"—a mathematical representation of a system's dynamics—is rarely perfect. Real-world environments introduce unpredictable disturbances such as wind gusts, changing friction, or unmodeled physical interactions. Traditional Iterative Learning Control (ILC) aims to improve performance by repeating tasks and learning from past errors. However, most ILC methods assume that disturbances are either negligible or follow a predictable Gaussian distribution.
This work addresses the challenge of non-stochastic control, where disturbances can be arbitrary, time-varying, or even adversarial. We introduce a framework that treats unmodeled dynamics as a regret minimization problem, leading to a controller that adapts faster and remains robust under significant model mismatch.
Defining Planning Regret
A core contribution of this work is the introduction of Planning Regret. Unlike standard metrics that evaluate absolute cost, planning regret compares the algorithm's cumulative performance across trials against the best possible strategy that could have been chosen in hindsight, given full knowledge of the disturbances.
By minimizing this regret, the controller seeks instance-wise optimality, adapting to the specific disturbances encountered in a particular rollout rather than optimizing for a generic average-case scenario.
The iGPC Algorithm
To minimize planning regret, we propose the Iterative Gradient Perturbation Controller (iGPC). The algorithm utilizes a nested structure based on Online Convex Optimization (OCO):
- Outer Loop (Across Rollouts): After each trial (rollout), the algorithm updates the base open-loop plan. It uses gradient descent to refine the trajectory for the next trial based on the total cost incurred.
- Inner Loop (During a Rollout): During execution, a Gradient Perturbation Controller (GPC) provides real-time closed-loop corrections. By observing disturbances as they occur, GPC adjusts control inputs on the fly using a "lookback window" of past errors to counteract structured noise.
This approach allows the system to "differentiate through the model," leveraging gradient-based updates that are significantly more efficient than derivative-free reinforcement learning methods.
Experimental Evaluation
We evaluated iGPC against standard benchmarks using JAX-based differentiable dynamics:
- Double Integrator: Tested against constant, increasing, and sinusoidal perturbations to verify basic robustness.
- Quadrotor with Wind: A non-linear 6D-state model navigating a dispersive force field. iGPC demonstrated superior resilience as wind magnitude increased.
- Reacher with Impulse: A 2-DOF robotic arm subjected to periodic physical impulses.
Key Results
- Theoretical Guarantees: We prove an average planning regret bound of $O(1/\sqrt{T} + 1/\sqrt{N})$, indicating that performance converges toward the optimum as the number of trials ($N$) and time horizon ($T$) increase.
- Convergence and Robustness: In non-linear experiments, iGPC reached lower costs faster than traditional ILC and Iterative Linear Quadratic Gaussian (ILQG) methods. The performance gap widened in favor of iGPC as model inaccuracies became more severe.
Conclusion and Future Work
The iGPC algorithm offers a robust alternative to traditional ILC by framing control as a regret minimization game against an adversarial environment. While current results are promising in fully observable simulations, future work will focus on extending this framework to partially observable settings and validating the algorithm on physical robotic hardware to assess sim-to-real transfer.
Summary updated