Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

Stefan Elfwing, Eiji Uchibe, Kenji DoyaarXiv 2017 · arXiv

This paper presents a compelling alternative to the dominant Deep Q-Network (DQN) paradigm in reinforcement learning (RL). By introducing two new activation functions—the Sigmoid-Weighted Linear Unit (SiLU) and its derivative (dSiLU)—and pairing them with "traditional" on-policy learning techniques, the authors demonstrate state-of-the-art performance in Tetris and significant improvements over DQN in the Atari 2600 domain.

TL;DR

The authors introduce SiLU and dSiLU activation functions, showing that when combined with "old-school" on-policy methods like Sarsa($\lambda$) and softmax action selection, they can outperform DQN without needing complex additions like experience replay or target networks. This approach achieved new state-of-the-art results in Tetris variants and significantly bested DQN in several Atari games.

What Problem Does This Paper Address?

Modern deep RL, popularized by DQN, relies heavily on off-policy Q-learning, experience replay buffers, and separate target networks to maintain stability. While successful, these components add complexity and can suffer from overestimation bias due to the "max" operator in Q-learning.

The authors ask: Can we achieve similar or better results by improving the neural network's internal architecture (activation functions) and returning to simpler, on-policy learning methods? They specifically target the limitations of standard activations like ReLU and Sigmoid in the context of RL function approximation.

Methods

The paper's primary contribution is the introduction of two activation functions:

  1. SiLU (Sigmoid-Weighted Linear Unit): Defined as $a_k(s) = z_k \sigma(z_k)$, where $\sigma$ is the sigmoid function. It acts as a smooth, "undershooting" version of ReLU. Its global minimum creates a "soft floor" that acts as an implicit regularizer, preventing weights from exploding.
  2. dSiLU: The derivative of SiLU. It looks like an "overshooting" sigmoid function, providing steeper gradients in certain ranges.
Figure 1: The activation functions of the SiLU and the ReLU (left panel), and the dSiLU and the sigmoid unit (right panel).

On the algorithmic side, the authors eschew the DQN "toolkit" in favor of:

  • On-policy learning: Using TD($\lambda$) and Sarsa($\lambda$).
  • Eligibility traces: To bridge the gap between immediate actions and delayed rewards.
  • Softmax action selection: Using Boltzmann distribution with hyperbolic annealing, which allows for more nuanced exploration than the standard $\epsilon$-greedy approach.

Datasets

The researchers evaluated their methods on three challenging benchmarks:

  • Stochastic SZ-Tetris: A simplified but notoriously difficult version of Tetris using only S and Z shapes, designed to test an agent's ability to handle "bad" states.
  • 10x10 Tetris: A smaller-board version of standard Tetris used to compare against other high-performing non-neural algorithms.
  • Atari 2600: A subset of 12 games (those starting with 'A' and 'B', such as Asterix, Asteroids, and Battle Zone) to provide an unbiased comparison against DQN and Double DQN.

Key Insights and Ideas

  • The Power of the "Soft Floor": The SiLU function's non-monotonic nature (it dips below zero before rising) provides a self-stabilizing property. This "soft floor" inhibits the learning of excessively large weights, which is often a source of instability in RL.
  • Bounded vs. Unbounded Units: In shallow networks (like those used for Tetris features), bounded units like dSiLU and Sigmoid outperformed unbounded units like SiLU and ReLU. However, in deep architectures, a hybrid approach (SiLU in convolutional layers, dSiLU in fully connected layers) proved most effective.
  • Softmax vs. $\epsilon$-greedy: A critical insight was that in games where a single mistake is fatal (like Asteroids or Tetris), $\epsilon$-greedy exploration is too dangerous because it treats the "worst" action and the "second-best" action with equal probability during exploration. Softmax exploration, by contrast, weights actions by their estimated value, leading to much safer learning.
  • Avoiding Overestimation: By using Sarsa($\lambda$) instead of Q-learning, the agents avoided the systematic overestimation of action values that plagues DQN, leading to more accurate value estimation.

Key Results

  • Tetris SOTA: The dSiLU agent achieved a new state-of-the-art score in SZ-Tetris, improving the previous best by 20%. In 10x10 Tetris, it reached an average of 4,900 points, surpassing the previous best of 4,200.
  • Atari Dominance: In the 12 Atari games tested, the SiLU-dSiLU agent achieved a mean DQN-normalized score of 218% (compared to 100% for DQN and 127% for Double DQN).
  • Specific Game Success: The performance was particularly massive in Asterix and Asteroids, where the proposed agent improved upon the second-best results by 562% and 552%, respectively.
Figure 3: Average Learning curves in stochastic SZ-Tetris for the three types of deep neural network agents. The figure shows the average scores over five separate runs, computed over every 1,000 episodes.

Limitations and Future Work

  • Training Time: The authors noted that training on a full-sized (20x10) Tetris board was computationally prohibitive for their current setup.
  • Failure in Breakout: The agent failed to "take off" in Breakout, suggesting that while this approach is powerful, it may not be a universal drop-in replacement for all game types without further tuning.
  • Integration: The paper suggests that these new activation functions and on-policy methods are not mutually exclusive with DQN's innovations. Future work could involve combining SiLU/dSiLU with dueling architectures or prioritized experience replay to see if the benefits are additive.

Summary updated

Abstract

In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection with simple annealing can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10$\times$10 board, using TD($λ$) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa($λ$) agent with SiLU and dSiLU hidden units.
BibTeX
@misc{elfwing2017sigmoid,
  author = {Stefan Elfwing and Eiji Uchibe and Kenji Doya},
  title = {{Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning}},
  year = {2017},
  eprint = {1702.03118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.03118},
}
Figure