Adaptive Batch Size for Safe Policy Gradients

Matteo Papini, Matteo Pirotta, Marcello Restelli

Introduction

In reinforcement learning (RL), policy gradient methods are powerful but inherently risky. Standard algorithms often suffer from "performance collapses" where a noisy gradient estimate leads to a detrimental policy update, causing the agent’s performance to plummet. In real-world scenarios like industrial robotics or financial management, such dips are unacceptable; systems must exhibit monotonic improvement over time.

While previous research has focused heavily on tuning the step size (how far to move in the gradient direction), the batch size (how many trajectories to sample to estimate that gradient) is often ignored or kept constant. This paper addresses that gap by introducing a framework to jointly optimize the step size and the batch size. By leveraging statistical concentration bounds, the authors provide a method that guarantees—with high probability—that every policy update will result in a performance improvement, making it a "safe" approach for practical applications.

Methodology

The authors build their safe update approach on two main pillars:

1. Non-Scalar Step Sizes

Instead of relying on a single global step size for all parameters, the authors propose using a diagonal matrix of step sizes. They mathematically demonstrate that the optimal strategy for maintaining a safe lower bound is a form of Greedy Coordinate Descent. In this approach, the algorithm updates only the single parameter component that exhibits the largest absolute gradient, minimizing the risk of compounding errors across multiple dimensions.

2. Adaptive Batch Size ($N$)

The paper defines a "cost-sensitive" performance measure: the expected policy improvement divided by the number of samples used. To maximize this safely, the authors employ statistical concentration bounds (such as Chebyshev, Hoeffding, and Empirical Bernstein) to estimate the error in the gradient.

This reveals a fundamental duality: if you use a small batch (high noise), you must use a tiny step size to remain safe. Conversely, if you use a large batch (low noise), you can confidently take a much larger step. By deriving a closed-form solution for the optimal batch size $N^*$, the authors show that when the batch size is optimized dynamically, the optimal step size actually becomes a constant value, greatly simplifying implementation.

Experimental Setup and Key Results

The proposed algorithms were evaluated on a Linear-Quadratic Gaussian (LQG) regulation problem. As a classic control theory benchmark where the optimal policy can be calculated analytically, LQG allows for precise measurements of how close the "safe" updates get to the theoretical optimum without the confounding variables of large-scale deep RL environments.

Estimator Choice and Bound Efficiency

The choice of gradient estimator plays a massive role in safe RL. The authors demonstrate that the G(PO)MDP gradient estimator is significantly better than the standard REINFORCE estimator because it has lower variance. Lower variance leads to tighter statistical bounds and, consequently, smaller required batch sizes.

Comparison of gradient estimators and statistical bounds.

Furthermore, using the Empirical Bernstein bound proved to be the most efficient approach. It allowed for smaller batch sizes and faster learning compared to the more pessimistic Chebyshev or Hoeffding bounds. The authors also noted that using the observed empirical range of gradients, rather than the theoretical maximum rigorous range, made the Hoeffding and Bernstein methods much more practical without sacrificing safety.

Safety via Conservatism

The algorithm is "conservative" by design, relying on a lower bound of performance improvement. If the gradient estimate is too noisy to guarantee an improvement, the algorithm effectively demands a larger batch size—choosing to wait for more data rather than making a risky move. As the agent approaches the optimal policy, the gradient naturally shrinks. To maintain the same confidence level, the batch size must increase, which the algorithm scales automatically.

Improvement ratios across different batch sizes and parameters.

In simulations, these adaptive methods consistently achieved an improvement ratio of 100% (or very close to it). This means the policy performance almost never decreased during training, successfully validating the theoretical safety guarantees.

Limitations and Future Work

While the theoretical guarantees are strong, the paper notes several limitations:

  • Gaussian Policy Constraint: The current mathematical derivations are specifically tailored to Gaussian policies. Extending this framework to other policy distributions (like Categorical distributions for discrete actions) is a necessary next step.
  • Conservatism vs. Speed: Because the method provides "high probability" guarantees, it can be slow. In practice, the batch sizes required by the rigorous bounds can become extremely large, stalling training.
  • Computational Overhead: Tighter bounds like Bernstein require estimating the variance of the gradient, adding computational complexity to each step.
  • Scaling to Deep RL: Future work is needed to evaluate how these adaptive batch sizes handle the high-dimensional parameter spaces typical of deep neural networks, where coordinate descent may become a bottleneck.

Summary updated

Abstract

Policy gradient methods are among the best Reinforcement Learning (RL) techniques to solve complex control problems. In real-world RL applications, it is common to have a good initial policy whose performance needs to be improved and it may not be acceptable to try bad policies during the learning process. Although several methods for choosing the step size exist, research paid less attention to determine the batch size, that is the number of samples used to estimate the gradient direction for each update of the policy parameters. In this paper, we propose a set of methods to jointly optimize the step and the batch sizes that guarantee (with high probability) to improve the policy performance after each update. Besides providing theoretical guarantees, we show numerical simulations to analyse the behaviour of our methods.
BibTeX
@inproceedings{papini2017adaptive,
  author = {Matteo Papini and Matteo Pirotta and Marcello Restelli},
  title = {{Adaptive Batch Size for Safe Policy Gradients}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=HyVPh_b_br},
}
Figure