Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL

Ghada Sokar, Johan Samir Obando Ceron, Aaron Courville, Hugo Larochelle, Pablo Samuel Castro

While Large Language Models reliably follow the "bigger is better" scaling law, Deep Reinforcement Learning (RL) has historically struggled with it. Traditionally, as you increase the size of a neural network for an RL agent, performance eventually plateaus or even degrades.

Recently, Soft Mixture of Experts (SoftMoE) architectures seemed to break this ceiling, allowing agents to scale and perform better as more "experts" (sub-networks) were added. However, the RL community didn't actually know why SoftMoE worked. Was it the conditional computation? The specialized experts? This paper conducts a rigorous "archaeological" dig into the SoftMoE architecture to reveal a surprising truth: the performance gains aren't coming from the "mixture of experts" at all. Instead, the secret sauce is tokenization.

Deconstructing SoftMoE

To find the true driver of SoftMoE's efficacy, the authors systematically ablated and isolated five key components using the Rainbow agent as their primary testbed:

  1. Weighted combination of tokens (how inputs are mixed).
  2. Expert specialization (whether experts learn specific tasks).
  3. Architectural dimensions (width and depth).
  4. Number of experts (scaling from 1 to 8).
  5. Tokenization (how the convolutional encoder's output is shaped).

The most novel approach introduced is the "Tokenized Baseline." Standard RL architectures typically destroy spatial information by "flattening" a 3D convolutional output (height × width × features) into a long 1D vector. Instead, the authors reshaped this output into a set of "tokens" (e.g., one token per spatial location) to preserve the spatial structure.

Different tokenization strategies compared to traditional flattening.

Key Insights: The Power of Spatial Structure

The study evaluated agents across a comprehensive suite of environments, including Atari 2600 (ALE), Procgen, and Continuous ALE (CALE). The findings fundamentally challenge how we understand MoEs in RL.

The "Aha!" Moment: Single Expert Suffices The most striking discovery is that a single, scaled-up expert using tokenization performs just as well as a complex mixture of multiple experts. The "mixture" part of MoE is largely redundant in current RL setups.

Performance comparison showing a single scaled expert (SoftMoE-1) matching the performance of multi-expert SoftMoE across different multipliers.

Flattening is the Enemy Tokenization preserves the spatial structure of the encoder outputs, which the authors show is crucial for the network to make sense of the environment. Simply adding tokenization to a standard "Rainbow-lite" baseline yielded statistically significant improvements without adding any extra parameters.

To prove that the network relies on this spatial grid and not just the raw features, the authors "shuffled" the tokens—destroying their spatial relationship but keeping the data identical. As expected, performance plummeted.

Expert Redundancy Because the spatial structure does the heavy lifting, the experts in SoftMoEs aren't actually specializing. In fact, the authors found that you can prune half the experts during training with almost zero impact on final performance.

These results held true across different encoder types (standard CNN and the more modern Impala/ResNet) and different algorithms (Rainbow and DER). Interestingly, the gains did not carry over to the original vanilla DQN, suggesting that tokenization might require the "categorical loss" found in distributional RL to be fully effective.

Limitations and Future Work

While this research successfully demystifies SoftMoE, it highlights a new problem: under-utilization. If a single expert is just as good as eight, we aren't yet capturing the true potential of Mixture of Experts in RL.

The authors suggest that future work should focus on forcing true specialization—finding ways to make experts actually learn different things. Additionally, maintaining "plasticity" using techniques like parameter resets could keep experts from becoming redundant or "dead" during the long training cycles typical of RL. Finally, further investigation is needed to understand exactly why tokenization fails to help the vanilla DQN architecture.

Summary updated

Abstract

The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have recently shown promise in mitigating this issue for online RL, the reasons behind their effectiveness remain largely unknown. In this work we provide an in-depth analysis identifying the key factors driving this performance gain. We discover the surprising result that tokenizing the encoder output, rather than the use of multiple experts, is what is behind the efficacy of SoftMoEs. Indeed, we demonstrate that even with an appropriately scaled single expert, we are able to maintain the performance gains, largely thanks to tokenization.
BibTeX
@inproceedings{sokar2025don,
  author = {Ghada Sokar and Johan Samir Obando Ceron and Aaron Courville and Hugo Larochelle and Pablo Samuel Castro},
  title = {{Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=8oCrlOaYcc},
}
Figure