While Large Language Models reliably follow the "bigger is better" scaling law, Deep Reinforcement Learning (RL) has historically struggled with it. Traditionally, as you increase the size of a neural network for an RL agent, performance eventually plateaus or even degrades.
Recently, Soft Mixture of Experts (SoftMoE) architectures seemed to break this ceiling, allowing agents to scale and perform better as more "experts" (sub-networks) were added. However, the RL community didn't actually know why SoftMoE worked. Was it the conditional computation? The specialized experts? This paper conducts a rigorous "archaeological" dig into the SoftMoE architecture to reveal a surprising truth: the performance gains aren't coming from the "mixture of experts" at all. Instead, the secret sauce is tokenization.
Deconstructing SoftMoE
To find the true driver of SoftMoE's efficacy, the authors systematically ablated and isolated five key components using the Rainbow agent as their primary testbed:
- Weighted combination of tokens (how inputs are mixed).
- Expert specialization (whether experts learn specific tasks).
- Architectural dimensions (width and depth).
- Number of experts (scaling from 1 to 8).
- Tokenization (how the convolutional encoder's output is shaped).
The most novel approach introduced is the "Tokenized Baseline." Standard RL architectures typically destroy spatial information by "flattening" a 3D convolutional output (height × width × features) into a long 1D vector. Instead, the authors reshaped this output into a set of "tokens" (e.g., one token per spatial location) to preserve the spatial structure.
Key Insights: The Power of Spatial Structure
The study evaluated agents across a comprehensive suite of environments, including Atari 2600 (ALE), Procgen, and Continuous ALE (CALE). The findings fundamentally challenge how we understand MoEs in RL.
The "Aha!" Moment: Single Expert Suffices The most striking discovery is that a single, scaled-up expert using tokenization performs just as well as a complex mixture of multiple experts. The "mixture" part of MoE is largely redundant in current RL setups.
Flattening is the Enemy Tokenization preserves the spatial structure of the encoder outputs, which the authors show is crucial for the network to make sense of the environment. Simply adding tokenization to a standard "Rainbow-lite" baseline yielded statistically significant improvements without adding any extra parameters.
To prove that the network relies on this spatial grid and not just the raw features, the authors "shuffled" the tokens—destroying their spatial relationship but keeping the data identical. As expected, performance plummeted.
Expert Redundancy Because the spatial structure does the heavy lifting, the experts in SoftMoEs aren't actually specializing. In fact, the authors found that you can prune half the experts during training with almost zero impact on final performance.
These results held true across different encoder types (standard CNN and the more modern Impala/ResNet) and different algorithms (Rainbow and DER). Interestingly, the gains did not carry over to the original vanilla DQN, suggesting that tokenization might require the "categorical loss" found in distributional RL to be fully effective.
Limitations and Future Work
While this research successfully demystifies SoftMoE, it highlights a new problem: under-utilization. If a single expert is just as good as eight, we aren't yet capturing the true potential of Mixture of Experts in RL.
The authors suggest that future work should focus on forcing true specialization—finding ways to make experts actually learn different things. Additionally, maintaining "plasticity" using techniques like parameter resets could keep experts from becoming redundant or "dead" during the long training cycles typical of RL. Finally, further investigation is needed to understand exactly why tokenization fails to help the vanilla DQN architecture.
Summary updated