NROWAN-DQN: A Stable Noisy Network with Noise Reduction and Online Weight Adjustment for Exploration

Shuai Han, Wenbo Zhou, Jing Liu, Shuai Lü

Abstract

Exploration is a fundamental challenge in Reinforcement Learning (RL), often requiring a delicate balance between discovering new strategies and maintaining policy stability. While NoisyNets improve upon traditional $\epsilon$-greedy methods by injecting noise into network parameters, they frequently suffer from high variance and slow convergence to a stable policy. This paper introduces NROWAN-DQN (Noise Reduction and Online Weight Adjustment NoisyNet-DQN), which incorporates a differentiable noise-reduction term into the loss function and an online weight adjustment mechanism. By explicitly minimizing output layer variance based on real-time performance, NROWAN-DQN achieves superior stability and higher scores across several OpenAI Gym benchmarks, including Cartpole and Pong.

Introduction

In Deep Reinforcement Learning, agents must explore their environment to find optimal policies. Traditional methods like $\epsilon$-greedy strategies rely on random action selection with a predefined decay probability, as shown in Figure 1. However, these methods often fail in complex or sparse-reward environments because the exploration schedule is independent of the agent's actual learning progress.

Standard DQN exploration using $\epsilon$-greedy strategy with a predefined decay schedule.

NoisyNets emerged as a more sophisticated alternative, injecting noise directly into the neural network's weights. This allows for state-dependent exploration that can be learned. Despite their advantages, standard NoisyNets often struggle to "settle down" into a stable policy because noise reduction is a passive byproduct of minimizing temporal difference (TD) error rather than an explicit objective. In action-sensitive environments like Cartpole, persistent noise can lead to catastrophic failures even after a good policy has been discovered.

Methodology: NROWAN-DQN

To address the instability of standard NoisyNets, researchers from Jilin University proposed NROWAN-DQN. This framework introduces two key innovations to the NoisyNet-DQN architecture to make stability a first-class citizen in the learning process.

Differentiable Noise Reduction

Unlike standard NoisyNets where noise decays spontaneously, NROWAN-DQN makes noise reduction an explicit part of the optimization process. The authors define a stability factor $D$ based on the variance ($\sigma$) of the network's output layer. This factor is added to the standard DQN loss function:

$$L^+ = L_{original} + k \cdot D$$

By making $D$ differentiable, the backpropagation process now explicitly pushes the output layer toward lower noise levels. This ensures that the agent's "hands" (the output layer) remain steady even if the "brain" (hidden layers) continues to explore internal representations through parameter noise.

NoisyNet-DQN architecture injecting parameter noise, with the last layer's standard deviation ($\sigma$) evolving during training.

Online Weight Adjustment

The coefficient $k$, which controls how aggressively the agent reduces noise, is adjusted online based on the agent's current reward ($r_t^+$) relative to the environment's known reward range.

  • Low Reward: $k$ remains small, allowing the agent to keep exploring and maintain high variance in its actions.
  • High Reward: $k$ increases, forcing the agent to stabilize its actions to "lock in" the successful policy and minimize performance variance.

Experimental Results

The algorithm was evaluated using the OpenAI Gym suite, focusing on benchmarks such as Cartpole-v1, Pong-v0, MountainCar, and Acrobot-v1. NROWAN-DQN consistently outperformed both standard DQN and NoisyNet-DQN across all tested domains.

Environment Metric NoisyNet-DQN NROWAN-DQN
Cartpole-v1 Average Score 164.96 187.04
Standard Deviation 31.56 13.99
Pong-v0 Average Score 17.95 18.81

In Cartpole, NROWAN-DQN achieved a nearly 3x improvement in stability, with the standard deviation dropping from 31.56 to 13.99. In the high-dimensional Pong environment, the agent reached higher scores with a tighter variance, proving the method scales effectively to visual inputs. The authors also found that the algorithm is relatively robust to the choice of the final weighting factor ($k_{final}$), with a value of 4.0 performing well across different learning rates.

Conclusion and Future Work

NROWAN-DQN demonstrates that treating policy stability as an explicit optimization goal leads to more reliable and higher-performing RL agents. By targeting the output layer for noise reduction and adjusting the reduction rate based on real-time rewards, the method effectively balances exploration and exploitation. Future work could involve integrating NROWAN-DQN with other DQN improvements like Prioritized Experience Replay or Distributional RL to create a more robust "Rainbow" agent.

Summary updated

BibTeX
@article{han2022nrowan,
  author = {Shuai Han and Wenbo Zhou and Jing Liu and Shuai Lü},
  title = {{NROWAN-DQN: A Stable Noisy Network with Noise Reduction and Online Weight Adjustment for Exploration}},
  journal = {Expert Systems with Applications},
  year = {2022},
  doi = {10.1016/j.eswa.2022.117343},
}
Figure