An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms

Li Wang, Sudun, Xingjian Zhang, Wenjun Wu, Lei Huang

Abstract

Batch Normalization (BN) has played a pivotal role in the success of deep learning by improving training stability, mitigating overfitting, and enabling more effective optimization. However, its adoption in deep reinforcement learning (DRL) has been limited due to the inherent non-i.i.d. nature of data and the dynamically shifting distributions induced by the agent's learning process. In this paper, we argue that, despite these challenges, BN retains unique advantages in DRL settings, particularly through its stochasticity and its ability to ease training. When applied appropriately, BN can adapt to evolving data distributions and enhance both convergence speed and final performance. To this end, we conduct a comprehensive empirical study on the use of BN in off-policy actor-critic algorithms, systematically analyzing how different training and evaluation modes impact performance. We further identify failure modes that lead to instability or divergence, analyze their underlying causes, and propose the Mode-Aware Batch Normalization (MA-BN) method with practical actionable recommendations for robust BN integration in DRL pipelines. We also empirically validate that, in RL settings, MA-BN accelerates and stabilizes training, broadens the effective learning rate range, enhances exploration, and reduces overall optimization difficulty.

Introduction

In supervised learning, Batch Normalization (BN) is a standard component that facilitates faster training and provides regularization. In Deep Reinforcement Learning (DRL), however, researchers have historically favored Layer Normalization (LN) or avoided normalization altogether. The primary obstacle is that RL violates the fundamental i.i.d. assumption of BN: agents collect data sequentially, leading to high correlation, and the learning process itself induces non-stationarity in the data distribution. When standard BN is applied naively to off-policy actor-critic methods, it often results in inaccurate value estimations and catastrophic divergence. This study investigates whether BN can be effectively integrated into DRL to leverage its unique benefits, such as inherent stochasticity for exploration and optimization landscape smoothing.

Addressing the Distribution Mismatch Challenge

The core difficulty in applying BN to DRL lies in the "distribution mismatch" that occurs during policy updates. In off-policy algorithms like SAC or DDPG, the critic evaluates the actor's current policy (Critic-I) using transitions sampled from a replay buffer. These transitions were generated by older versions of the policy. If BN is used in "Training Mode" during this step, it calculates statistics based on the current mini-batch from the buffer. However, the actor's current actions may differ significantly from the historical actions in the buffer, leading to biased Q-value estimates.

Figure 1: Analysis of rewards and Q-bias across different Critic-I modes in DeepMind Control Suite tasks.

As shown in Figure 1, using Training Mode for Critic-I (red lines) leads to significantly higher Q-bias and lower rewards compared to using Evaluation Mode (blue lines), which utilizes running averages of past statistics to provide a more stable baseline. This stability is crucial for providing accurate gradients to the actor.

Mode-Aware Batch Normalization (MA-BN)

Based on our systematic investigation, we propose Mode-Aware Batch Normalization (MA-BN). The key insight is that different components of the actor-critic architecture require different BN behaviors to balance adaptation and stability.

The MA-BN configuration is defined as follows:

  • Actor: Always operates in Training Mode (TT). This maximizes the stochasticity of the BN layers, which serves as a natural form of exploration noise, helping the agent visit diverse states.
  • Critic: Employs a hybrid ETT approach:
  • Evaluation Mode (E) for policy evaluation (Critic-I) to mitigate distribution mismatch and stabilize Q-value estimates.
  • Training Mode (T) for the critic update (Critic-II) and target value calculation (Critic-III), allowing the network to adapt its internal statistics to the current data distribution in the replay buffer.

Empirical Performance and Stability

We evaluated MA-BN across the DeepMind Control Suite (DMC), MuJoCo, and Atari benchmarks. MA-BN consistently outperformed both Layer Normalization and "No Normalization" baselines. In high-dimensional tasks like quadruped_walk, MA-BN achieved nearly double the reward of the baseline within the same number of environment steps.

Figure 2: Learning curves for the quadruped_walk task, illustrating the performance of MA-BN across varying replay buffer sizes.

Figure 2 highlights the robustness of the MA-BN approach (Critic-I E) compared to the unstable Training Mode (Critic-I T) across different replay buffer sizes. While larger buffers can sometimes exacerbate distribution shifts, the use of Evaluation Mode in Critic-I ensures consistent performance gains.

Beyond Performance: Exploration and Optimization

Beyond raw reward scores, MA-BN provides several qualitative advantages:

  1. Enhanced Exploration: In custom maze environments, MA-BN significantly increased state-space coverage compared to standard DDPG, confirming that BN's inherent noise aids discovery.
  2. Optimization Smoothing: MA-BN smooths the loss landscape so effectively that agents can be trained using simple Stochastic Gradient Descent (SGD), which typically fails in complex RL tasks compared to adaptive optimizers like Adam.
  3. Hyperparameter Robustness: MA-BN broadens the range of effective learning rates. While standard agents often diverge at higher learning rates (e.g., $10^{-3}$), MA-BN remains stable, reducing the need for extensive hyperparameter tuning.

Conclusion

While Batch Normalization has been historically difficult to stabilize in Reinforcement Learning, our investigation reveals that the issues stem from specific mode-selection failures rather than an inherent incompatibility. By adopting a Mode-Aware approach (MA-BN), practitioners can resolve distribution mismatches, leading to faster convergence, better exploration, and superior stability. MA-BN offers a robust alternative to Layer Normalization, particularly in off-policy settings where optimization difficulty and exploration are primary concerns.

Summary updated

Abstract

Batch Normalization (BN) has played a pivotal role in the success of deep learning by improving training stability, mitigating overfitting, and enabling more effective optimization. However, its adoption in deep reinforcement learning (DRL) has been limited due to the inherent non-i.i.d. nature of data and the dynamically shifting distributions induced by the agent's learning process. In this paper, we argue that, despite these challenges, BN retains unique advantages in DRL settings, particularly through its stochasticity and its ability to ease training. When applied appropriately, BN can adapt to evolving data distributions and enhance both convergence speed and final performance. To this end, we conduct a comprehensive empirical study on the use of BN in off-policy actor-critic algorithms, systematically analyzing how different training and evaluation modes impact performance. We further identify failure modes that lead to instability or divergence, analyze their underlying causes, and propose the Mode-Aware Batch Normalization (MA-BN) method with practical actionable recommendations for robust BN integration in DRL pipelines. We also empirically validate that, in RL settings, MA-BN accelerates and stabilizes training, broadens the effective learning rate range, enhances exploration, and reduces overall optimization difficulty. Our code is available at: https://github.com/monster476/ma-bn.git.
BibTeX
@misc{wang2025investigation,
  author = {Li Wang and Sudun and Xingjian Zhang and Wenjun Wu and Lei Huang},
  title = {{An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms}},
  year = {2025},
  eprint = {2509.23750},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23750},
}
Figure