Abstract
Automating the parameter tuning of industrial controllers remains a significant challenge due to the non-differentiable nature of classic control laws and the high dimensionality of modern control tasks. This paper introduces Zeroth-Order Actor-Critic (ZOAC), a reinforcement learning framework designed to optimize non-differentiable controllers like PID and Model Predictive Control (MPC). By combining derivative-free optimization with the temporal efficiency of actor-critic architectures, ZOAC achieves faster and more effective tuning than traditional methods such as Bayesian Optimization (BO) or Evolutionary Strategies (ES). Evaluation on autonomous driving tasks demonstrates that ZOAC scales effectively to high-dimensional parameter spaces while maintaining superior sample efficiency.
1. Introduction
In industrial control, "tuning"—the process of finding optimal parameters for a controller—is frequently a manual, iterative process often described as "black magic." While automation is highly desirable, existing methods face a fundamental trade-off. Derivative-Free Optimization (DFO) methods like Bayesian Optimization work well for low-dimensional problems but fail to scale. Conversely, Evolutionary Algorithms (EAs) scale better but treat entire episodes as single data points, leading to high sample complexity. Standard Reinforcement Learning (RL) is highly efficient but typically requires the controller to be a differentiable neural network, making it incompatible with classic, non-differentiable controllers like MPC.
ZOAC bridges this gap by creating a method that is as versatile as a black-box optimizer but as efficient as a modern RL agent.
2. Methods
The ZOAC framework treats controller tuning as a Markov Decision Process (MDP) and introduces several key innovations to enable efficient optimization without requiring gradients of the controller itself.
2.1 Timestep-wise Parameter Perturbation
Unlike standard Evolution Strategies that modify parameters once per episode, ZOAC injects small Gaussian noise into the controller parameters at every timestep. This approach leads to more diverse exploration and smoother trajectories compared to adding noise directly to control actions, which often results in high-frequency jitter that physical systems naturally filter out.
2.2 Zeroth-Order Policy Gradient
Because the controller is non-differentiable, the authors derive a new policy gradient theorem based on zeroth-order (function value only) information. This allows the framework to estimate a gradient direction for the parameters based on the observed rewards without needing an analytical derivative of the control law.
2.3 Actor-Critic Integration and Credit Assignment
ZOAC utilizes a "Critic" (Value Function) to perform temporal credit assignment. Instead of evaluating the total reward of an episode, the "Actor" (the controller being tuned) is updated based on the Advantage function using Generalized Advantage Estimation (GAE). This allows the system to identify which specific moments in a trajectory contributed to success or failure, providing a much denser training signal than traditional black-box tuners.
2.4 Parameter Mapping
To handle varying scales across different controller types, ZOAC employs linear mapping for gains and logarithmic mapping for weight matrices. This normalizes the search space to a consistent $[-1, 1]^d$ hypercube.
3. Experimental Setup
The method was evaluated using two high-fidelity simulation environments focused on autonomous driving:
- PID-ACC (Adaptive Cruise Control): Tuning 4 parameters ($k, K_p, K_i, K_d$) to maintain distance from a lead vehicle while optimizing for comfort and fuel efficiency.
- MPC-Tracking: A high-dimensional task involving 19 parameters, including 6 physical model parameters (e.g., tire stiffness and mass) and 13 cost-function weights. This represents a complex "Identification for Control" (I4C) problem.
4. Key Results and Insights
4.1 Scalability and Efficiency
In the 19-parameter MPC task, ZOAC significantly outperformed both BO and CMA-ES. While CMA-ES often converged to local optima and BO suffered from high computational overhead, ZOAC reached optimal performance significantly faster. Specifically, ZOAC required approximately 6,500 seconds of tuning time, compared to 27,000 seconds for Bayesian Optimization.
4.2 The Role of Temporal Credit Assignment
The primary advantage of ZOAC lies in its use of the Critic. Traditional tuners are "blind" to intra-episode dynamics. By formulating tuning as an RL problem, ZOAC uses the Critic to provide a dense training signal, which drastically reduces the number of samples required to find optimal parameters.
4.3 Identification for Control (I4C)
The results reinforce the principle that a "perfect" model is not always necessary for superior control. In the MPC experiments, adding a penalty for model inaccuracy improved predictive power but did not necessarily improve control performance. A model tuned specifically for the control task often outperforms a model tuned in isolation for predictive accuracy.
5. Limitations and Future Work
Despite its empirical success, the authors note that the theoretical asymptotic convergence of two-time-scale actor-critic methods using neural critics remains an open challenge. Additionally, because the current work is simulation-based, future research must address "safe exploration" to ensure that parameter perturbations do not lead to dangerous maneuvers in real-world physical systems. Finally, extending the framework to handle discrete parameter spaces or switching control laws remains a promising direction for future development.
6. Conclusion
ZOAC represents a significant step forward in automated controller tuning. By integrating zeroth-order optimization with the temporal advantages of actor-critic RL, it provides a scalable, efficient, and versatile tool for optimizing the complex controllers used in modern industrial and automotive applications.
Summary updated