RLBenchNet: The Right Network for the Right Reinforcement Learning Task

Ivan Smirnov, Shangding GuarXiv 2025 · DOI · arXiv

Introduction

In Reinforcement Learning (RL), research typically prioritizes algorithmic innovation (e.g., PPO, SAC) while treating the underlying neural network architecture as a secondary implementation detail. However, the "backbone" of an agent is fundamental to its capacity for handling partial observability, temporal credit assignment, and computational efficiency. RLBenchNet provides a systematic benchmark across diverse architectures—Multi-Layer Perceptrons (MLP), Recurrent Neural Networks (LSTM/GRU), Transformers (TrXL/GTrXL), and State-Space Models (Mamba/Mamba-2)—to identify the optimal "brain" for specific RL challenges.

Methodology

To ensure a rigorous comparison, all evaluations utilize Proximal Policy Optimization (PPO). Architectures are standardized by adjusting network depth and width to maintain approximately equal parameter counts. The benchmark includes:

  • MLP: Standard feed-forward baseline.
  • LSTM & GRU: Traditional recurrent models for sequential data.
  • Transformer-XL (TrXL) & Gated Transformer-XL (GTrXL): Attention-based models for long-range dependencies.
  • Mamba & Mamba-2: Modern State-Space Models (SSMs) designed for high throughput and efficient long-sequence processing.

Experimental Setup

The study spans four distinct environment categories:

  1. Continuous Control (MuJoCo): Walker2d, HalfCheetah, and Hopper (fully observable).
  2. Discrete Decision-Making (Atari): Pong and Breakout.
  3. Partially Observable MDPs (POMDPs): Masked versions of CartPole and LunarLander (velocity hidden).
  4. Memory-Intensive Tasks (MiniGrid): DoorKey (credit assignment) and Memory-S11 (long-term retention).

Results and Discussion

Performance in Markovian Environments

For tasks with smooth dynamics and full observability, such as the MuJoCo suite, simpler architectures often prove superior. As shown in Figure 1, standard MLPs frequently match or exceed the performance of more complex recurrent or attention-based models. In these scenarios, the overhead of managing hidden states or attention masks can impede learning speed without providing a commensurate increase in final reward.

Figure 1: Performance comparison on continuous control MuJoCo tasks, where simpler architectures like MLPs remain highly competitive.

Overcoming the "Memory Wall"

The most significant differentiation occurs in memory-intensive environments. While most architectures struggle with long-term information retention, Mamba-2 and GTrXL demonstrate a clear advantage. In the MiniGrid Memory-S11 task, these models were the only ones to achieve near-optimal rewards. Notably, Mamba-2 achieved this performance while using 8x less GPU memory than Transformer-XL (0.217 GB vs. 1.765 GB), highlighting its potential as a scalable alternative for long-horizon RL.

Figure 6: Performance on memory-intensive MiniGrid tasks. Mamba-2 and GTrXL successfully navigate the 'Memory Wall' in the Memory-S11 environment.

Computational Efficiency

Mamba models emerged as the most efficient, reaching an average throughput of 2734 steps per second (SPS). This represents a 4.5x increase over LSTMs and a 3.9x increase over GRUs. This efficiency, combined with low memory footprints, makes SSMs particularly attractive for resource-constrained or high-throughput training scenarios.

Conclusion

RLBenchNet demonstrates that there is no "one-size-fits-all" architecture for RL. MLPs are optimal for fully observable tasks due to their simplicity and speed. For POMDPs requiring short-term memory, GRUs and LSTMs remain robust choices. However, for complex, long-horizon tasks, Mamba-2 offers a breakthrough, providing Transformer-level performance with significantly reduced computational overhead. Future work will explore hybrid architectures and the impact of these backbones on off-policy algorithms.

Summary updated

Abstract

Reinforcement learning (RL) has seen significant advancements through the application of various neural network architectures. In this study, we systematically investigate the performance of several neural networks in RL tasks, including Long Short-Term Memory (LSTM), Multi-Layer Perceptron (MLP), Mamba/Mamba-2, Transformer-XL, Gated Transformer-XL, and Gated Recurrent Unit (GRU). Through comprehensive evaluation across continuous control, discrete decision-making, and memory-based environments, we identify architecture-specific strengths and limitations. Our results reveal that: (1) MLPs excel in fully observable continuous control tasks, providing an optimal balance of performance and efficiency; (2) recurrent architectures like LSTM and GRU offer robust performance in partially observable environments with moderate memory requirements; (3) Mamba models achieve a 4.5x higher throughput compared to LSTM and a 3.9x increase over GRU, all while maintaining comparable performance; and (4) only Transformer-XL, Gated Transformer-XL, and Mamba-2 successfully solve the most challenging memory-intensive tasks, with Mamba-2 requiring 8x less memory than Transformer-XL. These findings provide insights for researchers and practitioners, enabling more informed architecture selection based on specific task characteristics and computational constraints. Code is available at: https://github.com/SafeRL-Lab/RLBenchNet
BibTeX
@misc{smirnov2025rlbenchnet,
  author = {Ivan Smirnov and Shangding Gu},
  title = {{RLBenchNet: The Right Network for the Right Reinforcement Learning Task}},
  year = {2025},
  eprint = {2505.15040},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.15040},
}
Figure