TL;DR
While Batch Normalization (BN) is standard in supervised learning, deep reinforcement learning (RL) practitioners usually avoid it in favor of Layer Normalization due to non-stationary, temporally correlated data. This paper shows that BN’s failures in off-policy actor-critic RL stem from improper toggling between training (batch stats) and evaluation (running stats) modes across update steps. By introducing Mode-Aware Batch Normalization (MA-BN), the authors demonstrate that properly configured BN outperforms LayerNorm, improves exploration via batch stochasticity, widens usable learning rate ranges, and even makes pure SGD viable.
What Problem Does This Paper Address?
In deep RL, data distributions continuously shift as an agent interacts with its environment and updates its policy. Because Batch Normalization relies on stable batch statistics to estimate running population statistics, standard implementations often suffer catastrophic divergence or severe Q-value bias in RL pipelines.
Consequently, modern algorithms routinely fall back on Layer Normalization (LN), which normalizes across features rather than mini-batch samples. However, LN misses out on BN's key advantages: gradient smoothing and mini-batch stochasticity (which can aid exploration and regularization). While isolated methods like CrossQ have used BN variants, no prior work systematically studied when a network should use batch statistics versus running statistics during the distinct steps of off-policy actor-critic updates.
Methods
The authors decompose standard off-policy actor-critic updates into five distinct forward passes:
- Actor-I: Policy forward pass producing actions for the actor update ($\pi(s_t)$).
- Actor-II: Policy forward pass selecting next-step actions for critic targets ($\pi(s_{t+1})$).
- Critic-I: Critic forward pass evaluating the actor's suggested actions ($Q(s_t, \pi(s_t))$).
- Critic-II: Critic forward pass on replay buffer transitions to compute Bellman error ($Q(s_t, a_t)$).
- Critic-III: Target critic forward pass computing target values ($Q_{\bar{\theta}}(s_{t+1}, a_{t+1})$).
Each forward pass can run in either Training mode (T) (using mini-batch statistics) or Evaluation mode (E) (using tracked running statistics).
Through theoretical and empirical analysis, the authors propose Mode-Aware Batch Normalization (MA-BN):
- Critic configuration — ETT: Critic-I operates in Eval mode (E), while Critic-II and Critic-III operate in Train mode (T).
- Actor configuration — TT: Both Actor-I and Actor-II operate in Train mode (T) to maximize beneficial stochasticity.
Datasets
The authors evaluated MA-BN across diverse environments:
- DeepMind Control Suite (DMC): 8 continuous control tasks (hopper_hop, quadruped_run, quadruped_walk, reacher_easy, reacher_hard, acrobot_swingup, cartpole_swingup_sparse, and reach_duplo).
- MuJoCo: Continuous control benchmarks (Humanoid-v4, Walker2d-v4).
- Navigation Mazes: Standard D4RL tasks (maze2d-umaze, maze2d-medium) and a custom continuous 2D maze with obstacle and death-zone navigation.
- Atari 2600: Discrete control benchmarks (Pong, Alien, Freeway).
- Synthetic/Toy Problem: A discrete-time Linear Quadratic Regulator (LQR) environment for exact analytic tracking of Q-function bias.
Key Insights and Ideas
- The Critic-I Failure Mechanism: Setting Critic-I to Training mode (Critic-I T) causes drastic failure. Why? In Critic-II, the critic trains on replay buffer actions. In Critic-I, actions are sampled from the current actor policy. Calculating batch statistics exclusively over current actor actions causes a severe distribution shift from the replay buffer distribution. This produces massive Q-value overestimation bias and feedback loops.
- Proof via Data Mixing: When Critic-I is run in Training mode but augmented with auxiliary samples from the replay buffer (solely for computing BN statistics, without computing their gradients), training stabilizes and matches Eval mode performance.
- Beneficial Exploration via Actor Stochasticity: Keeping the actor in Train mode (TT) injects mini-batch sampling noise into action selection, acting as a natural, complementary exploration mechanism that improves state coverage in navigation environments without explicit exploration bonuses.
- Optimizer Flexibility: Because BN conditions the loss landscape, MA-BN substantially widens the acceptable learning rate window and enables effective training using basic SGD, mitigating the memory overhead of adaptive optimizers like Adam.
Key Results
- Critic-I Mode Impact (DRQv2 on DMC):
- quadruped_run: Critic-I E achieves a converged return of $642.40 \pm 79.44$ vs. $116.54 \pm 65.19$ for Critic-I T.
- quadruped_walk: Critic-I E achieves $890.86 \pm 31.63$ vs. $133.98 \pm 68.55$ for Critic-I T.
- hopper_hop: Critic-I E achieves $278.55 \pm 154.46$ vs. $77.02 \pm 19.73$ for Critic-I T.
- Comparison to Alternatives: On continuous control benchmarks (DMC), DRQv2 equipped with MA-BN consistently outpaces both LayerNorm (LN) and unnormalized baselines (Origin) in sample efficiency and final return.
- Exploration Coverage: In 2D maze environments, DDPG and SAC with MA-BN exhibited higher state-space coverage and faster goal convergence compared to unnormalized and LN counterparts.
Limitations and Future Work
- Actor Mode Variability: While the TT configuration is optimal for DMC, empirical performance on MuJoCo SAC showed ET performing slightly better than TT, indicating optimal actor BN modes can still exhibit minor task/algorithm dependencies.
- Replay Buffer Sensitivity: While Critic-I E handles distribution mismatch, in environments like quadruped_run, excessively small replay buffers still hurt performance due to strong off-policy sample correlations.
- Scope: Further work is needed to validate MA-BN on large-scale foundation models in RL and multi-task/offline-to-online transfer benchmarks.
Summary updated