Introduction
In Reinforcement Learning (RL), research typically prioritizes algorithmic innovation (e.g., PPO, SAC) while treating the underlying neural network architecture as a secondary implementation detail. However, the "backbone" of an agent is fundamental to its capacity for handling partial observability, temporal credit assignment, and computational efficiency. RLBenchNet provides a systematic benchmark across diverse architectures—Multi-Layer Perceptrons (MLP), Recurrent Neural Networks (LSTM/GRU), Transformers (TrXL/GTrXL), and State-Space Models (Mamba/Mamba-2)—to identify the optimal "brain" for specific RL challenges.
Methodology
To ensure a rigorous comparison, all evaluations utilize Proximal Policy Optimization (PPO). Architectures are standardized by adjusting network depth and width to maintain approximately equal parameter counts. The benchmark includes:
- MLP: Standard feed-forward baseline.
- LSTM & GRU: Traditional recurrent models for sequential data.
- Transformer-XL (TrXL) & Gated Transformer-XL (GTrXL): Attention-based models for long-range dependencies.
- Mamba & Mamba-2: Modern State-Space Models (SSMs) designed for high throughput and efficient long-sequence processing.
Experimental Setup
The study spans four distinct environment categories:
- Continuous Control (MuJoCo): Walker2d, HalfCheetah, and Hopper (fully observable).
- Discrete Decision-Making (Atari): Pong and Breakout.
- Partially Observable MDPs (POMDPs): Masked versions of CartPole and LunarLander (velocity hidden).
- Memory-Intensive Tasks (MiniGrid): DoorKey (credit assignment) and Memory-S11 (long-term retention).
Results and Discussion
Performance in Markovian Environments
For tasks with smooth dynamics and full observability, such as the MuJoCo suite, simpler architectures often prove superior. As shown in Figure 1, standard MLPs frequently match or exceed the performance of more complex recurrent or attention-based models. In these scenarios, the overhead of managing hidden states or attention masks can impede learning speed without providing a commensurate increase in final reward.
Overcoming the "Memory Wall"
The most significant differentiation occurs in memory-intensive environments. While most architectures struggle with long-term information retention, Mamba-2 and GTrXL demonstrate a clear advantage. In the MiniGrid Memory-S11 task, these models were the only ones to achieve near-optimal rewards. Notably, Mamba-2 achieved this performance while using 8x less GPU memory than Transformer-XL (0.217 GB vs. 1.765 GB), highlighting its potential as a scalable alternative for long-horizon RL.
Computational Efficiency
Mamba models emerged as the most efficient, reaching an average throughput of 2734 steps per second (SPS). This represents a 4.5x increase over LSTMs and a 3.9x increase over GRUs. This efficiency, combined with low memory footprints, makes SSMs particularly attractive for resource-constrained or high-throughput training scenarios.
Conclusion
RLBenchNet demonstrates that there is no "one-size-fits-all" architecture for RL. MLPs are optimal for fully observable tasks due to their simplicity and speed. For POMDPs requiring short-term memory, GRUs and LSTMs remain robust choices. However, for complex, long-horizon tasks, Mamba-2 offers a breakthrough, providing Transformer-level performance with significantly reduced computational overhead. Future work will explore hybrid architectures and the impact of these backbones on off-policy algorithms.
Summary updated