Pricing the Noise Floor: Coupling Exploration Noise and Batch Size in Policy Gradients

Policy gradient methods train a stochastic policy, a network that outputs a distribution over actions, by nudging it toward actions that earned high reward. As training succeeds, the policy grows less random. Recent analysis shows that this success damages the learning signal. Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator proves that for a Gaussian policy with standard deviation $\sigma$, the ratio of gradient noise to true gradient grows as $\sigma^{-2}$ once the optimal policy is deterministic. So the closer the agent gets to mastery, the more its updates are pure noise. The same paper shows that noise-dominated updates tend to shrink $\sigma$ further, which creates a feedback loop that can collapse returns. This is a gradient noise floor, the region where stochastic optimization stalls and oscillates instead of converging, and the agent creates it for itself. It also wastes samples: environment interactions keep being spent on updates that carry almost no information.

We propose to treat the policy's exploration noise $\sigma$ and the number of trajectories per update $N$ as two controls of a single quantity, the measured noise-to-signal ratio of each update. A controller holds that ratio near a fixed target. Whenever it rises, the controller chooses the cheaper of two fixes, measured in environment samples per unit of expected improvement: collect more trajectories, or temporarily stop the policy from becoming less random. The claim is that most late-stage instability and sample waste in on-policy policy gradients comes from letting $\sigma$ and $N$ drift independently. Coupling them should reach a given level of suboptimality with fewer samples than either fix alone, and fewer than existing adaptive batch schemes that ignore $\sigma$.

Core Claim

Han and Yang's analysis implies a simple accounting rule. For a batch of $N$ trajectories, the noise-to-signal ratio behaves approximately as

$$\mathrm{NSR}_N \approx \frac{c}{N\,\sigma^{2}},$$

where $c$ is a problem-dependent constant. Han and Yang show the single-trajectory ratio scales as $\sigma^{-2}$, and averaging $N$ independent trajectories divides the variance by $N$. In linear-quadratic control, the extra cost caused by exploration noise grows roughly with $\sigma^2$. Reaching suboptimality $\varepsilon$ therefore needs $\sigma^2 \sim \varepsilon$. Holding the ratio at a target $\kappa$ then needs $N \gtrsim c/(\kappa\varepsilon)$ trajectories per update.

This gives a predicted price for each further gain in precision. It also says the schedule should not be "grow the batch" or "keep exploring" but a specific trade between the two. The new element is an algorithm that measures this ratio online in deep networks and acts on it through both controls. It is not a fixed batch schedule, a behavioral heuristic, or a worst-case confidence bound.

Relation to Prior Work

Three works motivate or constrain the design. Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator supplies the mechanism: the $\sigma^{-2}$ blow-up, its persistence under imperfect value baselines, and the collapse loop. It stops short of proposing a schedule, and we build one on it. Adaptive Batch Size for Safe Policy Gradients jointly picks step and batch size from concentration inequalities so that each update improves the policy with high probability. Its bounds are conservative, demand very large batches near the optimum, and were tested only on linear controllers. We keep its useful idea of a samples-per-improvement objective but replace worst-case bounds with a measured ratio. We also add $\sigma$ as a second control, which Pirotta et al. treat as fixed. Signal-to-Noise Ratio Analysis of Policy Gradient Algorithms earlier showed that the shape of exploration noise sets the signal-to-noise ratio. That supports treating exploration noise as an optimization control and not only an exploration choice.

The closest existing method is Scalable Reinforcement Learning via Adaptive Batch Scaling. It grows batch size as a cheap measure of policy change falls: the fraction of states whose action changes between updates. It argues that late training needs large batches to lower the noise floor, while early training benefits from small batches. Our proposal differs in two ways. First, the trigger is the gradient noise-to-signal ratio, the quantity the theory says matters, not how much the policy's behavior changes. Second, batch growth is only one of two responses. Park's divergence signal can fall for two different reasons: the policy has converged, or $\sigma$ has collapsed under noise. These call for different actions, and the divergence signal cannot tell them apart.

Two works constrain the scope. Batch size-invariance for policy optimization shows that PPO's learning dynamics stay the same across batch sizes only in the small-batch regime, where signal-to-noise is near 1. That is exactly the regime our controller aims to stay in, and we adopt their Adam step-size scaling rule ($1/\sqrt{c}$) when $N$ changes. The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning finds that smaller batches help value-based agents that use multi-step returns. This might seem to contradict any batch-growth method. We do not claim larger batches are always better: our controller grows $N$ only when the measured ratio exceeds target, and that work concerns value-based learning. We list multi-step value learning as out of scope rather than claim the method transfers.

Research Program

The controller needs a cheap estimate of the noise-to-signal ratio in deep networks. We split each batch into two halves and compute the gradient on each half and on the full batch. The squared norm of a minibatch gradient equals the true squared gradient plus noise trace divided by batch size. Two batch sizes therefore give unbiased estimates of both the signal and the noise trace. The cost is one extra backward pass on half the data. We smooth both estimates with an exponential moving average and report the overhead, both on its own and next to Park's single forward pass.

The controller runs at each update. If the estimated $\mathrm{NSR}_N$ exceeds $\kappa$, it compares two costs. Doubling $N$ costs $N$ extra trajectories per update. Freezing $\sigma$ costs the expected return lost by not reducing exploration, which we estimate from the recent slope of return against $\log\sigma$. It applies the cheaper fix. Freezing means projecting out the noise-driven component of the gradient on the log-standard-deviation parameters, which targets the collapse loop Han and Yang describe. When the ratio falls well below $\kappa$, $N$ shrinks back, which keeps early-training batches small as Park recommends.

On the analysis side, we derive the predicted samples-to-$\varepsilon$ scaling for finite-horizon linear-quadratic-Gaussian (LQG) systems, using Han and Yang's exact ratio expressions. We compare it against the predictions for schedules that only grow batch size and schedules that only clamp $\sigma$.

Baselines

The naive reference is PPO with fixed batch size and an annealed, learned $\sigma$, tuned on the same budget as everything else. It shows how much the noise floor costs in practice. We expect collapse events late in training, as Han and Yang observed on Hopper-v4 and HalfCheetah-v4.

The strongest existing competitor is Adaptive Batch Scaling from Scalable Reinforcement Learning via Adaptive Batch Scaling. It is the only method in the file validated on deep networks that adapts batch size to late-stage noise. Park reports "up to a 190% improvement where fixed large batches caused models to fail" on Atari. The file contains no per-task MuJoCo or LQG return for it (no reported result found on Lacuna for Adaptive Batch Scaling on Hopper-v4 or HalfCheetah-v4). We will therefore reimplement it in the same PPO codebase. It gets the same environment-sample budget and the same number of tuning trials for its divergence thresholds as we get for $\kappa$. Beating it means a statistically reliable improvement in area under the return-versus-samples curve and in fewer seeds that collapse, across at least most tasks.

On LQG, the reference is the concentration-bound method of Adaptive Batch Size for Safe Policy Gradients, which reports "an improvement ratio of 100% (or very close to it)". We do not expect to beat it on monotonic improvement. The test is whether we reach the same suboptimality with fewer trajectories while keeping the fraction of improving updates near that level.

Two ablations isolate the key ingredient, the coupling of the two controls. A batch-only variant uses our ratio estimate to grow $N$ but leaves $\sigma$ alone. This is a gradient-noise-scale rule specialized to policy gradients. A $\sigma$-only variant freezes $\sigma$ at fixed $N$. If either ablation matches the full method, the coupling claim fails.

Why not just…?

Why not just put a floor on $\sigma$ or add a tuned entropy bonus? This is the most serious threat. Han and Yang note that entropy regularization reduces the blow-up, and a well-chosen floor removes the singularity directly. The cost is that the policy can never become deterministic, so a fixed floor caps final precision at the floor's exploration cost. We will run a sweep of fixed floors and entropy coefficients, with tuning trials equal to ours, and pick the best one for each task. If the best fixed floor matches us at every sample budget, our contribution shrinks to automatic tuning of that floor, and we will report it as such. The coupled controller should win when the budget is large enough that the floor's precision cap binds. The LQG experiment shows where that crossover falls.

Why not just decay the learning rate? Decay shrinks the noise ball. But Two Time-scale Off-Policy TD Learning: Non-asymptotic Analysis over Markovian Samples shows that diminishing schedules converge sluggishly compared with constant steps. Decay also does nothing about the noise pushing $\sigma$ downward; it only slows that collapse. We include tuned cosine decay as a baseline.

Why not just use Park's behavioral divergence signal? It is cheaper. Our diagnostic experiment will log both signals and check whether divergence falls during episodes of $\sigma$ collapse, when the right action is to freeze $\sigma$, not to enlarge the batch. If the two signals select the same actions, the simpler signal wins.

Proposed Experiments

The first testbed is finite-horizon LQG and the Double Integrator, used by Han and Yang. There the exact ratio is known in closed form, so we can check the estimator's accuracy against it. We will also test the predicted relationship: per-update trajectories should scale as $1/\varepsilon$ at fixed ratio target. The decisive check is whether measured samples-to-$\varepsilon$ curves for the coupled controller, the batch-only variant, and the $\sigma$-only variant fall in the predicted order with the predicted slopes.

The second testbed is MuJoCo Hopper-v4 and HalfCheetah-v4 with PPO and Gaussian policies, plus a broader set of DeepMind Control tasks. We use at least ten seeds and report three things: interquartile-mean return against environment samples, the fraction of seeds with a late-training return drop above a fixed threshold, and the final $\sigma$.

The third testbed checks how far the idea extends beyond its theory. Han and Yang's theory covers only Gaussian policies. We apply the batch-only ratio controller, with the entropy of a categorical policy in place of $\sigma$, to Atari-10 under Park's protocol, so the comparison runs on Park's own benchmark. This part is a stated extrapolation, not a claim the theory supports.

The proposal fails if, in the second testbed, a tuned fixed $\sigma$ floor or Adaptive Batch Scaling matches the coupled controller on both sample efficiency and collapse rate.