ScaleMoE: Mixture-of-Experts for Scalable Continuous Control in Actor-Critic Reinforcement Learning

Yi Ma, Chenjun Xiao, Hongyao Tang, Yaodong Yang, Jinyi Liu, Jing Liang, Jiye LiangICML 2026 · OpenReview

In the domains of Computer Vision and Natural Language Processing, the recipe for success has largely been straightforward: make the model bigger. However, Deep Reinforcement Learning (RL) has famously struggled to replicate this "scaling law." Historically, simply adding more layers or neurons to an actor-critic network leads to training instability, overfitting, and a rapid plateau in performance.

A new architecture, ScaleMoE, proposes a compelling solution to this bottleneck by introducing Mixture-of-Experts (MoE) to continuous control. By utilizing sparse, specialized sub-networks instead of a single massive monolithic block, ScaleMoE demonstrates that RL can finally achieve the positive scaling trajectories seen in Large Language Models.

The Bottleneck in RL Scaling

In deep RL, "bigger" rarely equates to "better." Standard monolithic architectures—where every parameter is activated for every single input—suffer from two primary issues when scaled up:

  1. Instability and Loss of Plasticity: Large networks are notoriously difficult to optimize in the non-stationary environments typical of RL. This often leads to a high ratio of "dormant neurons"—units that effectively die and stop contributing to the learning process.
  2. Diminishing Returns: Even with recent architectures like SimBa and BRC that introduce clever inductive biases, performance typically plateaus or degrades once the network reaches a certain capacity.

The core argument of ScaleMoE is that the monolithic structure itself is the limiting factor. To increase total model capacity effectively, the architecture must avoid forcing every neuron to process every state.

The ScaleMoE Architecture

ScaleMoE replaces the standard hidden layers of both the actor and the critic with a set of $N$ "experts." A learnable gating network evaluates the current state and dynamically routes the input to the top $K$ experts best suited to handle it (where $K < N$).

The architecture explores two primary integration strategies:

  • Output-Level Gating (Late Fusion): Each expert independently predicts a final output, such as a Q-value or a policy mean/variance. The gating network then blends these individual predictions into a final result.
  • Feature-Level Gating (Early Fusion): Experts produce internal representations (features) that are fused into a single vector. This combined representation is then passed through a shared "head" to generate the final output.

To prevent "expert collapse"—a common failure mode where the gating network over-relies on a few experts and ignores the rest—ScaleMoE incorporates load-balancing and importance losses. These auxiliary objectives ensure the workload is distributed evenly across the entire expert pool.

Key Insights and Performance Gains

The authors rigorously evaluated ScaleMoE across challenging continuous control benchmarks, including DeepMind Control Suite (DMC) Hard Tasks, MetaWorld (50 distinct robotic manipulation tasks), and HumanoidBench (20 complex tasks for a simulated Unitree H1 humanoid).

1. Joint Actor-Critic Scaling

Unlike previous attempts that applied MoE exclusively to the policy, ScaleMoE scales both the actor and the critic. A high-capacity actor is ultimately ineffective if the critic cannot provide a stable, high-quality signal to train it.

2. Breaking the Performance Plateau

ScaleMoE exhibits a clear, upward scaling trend. As the number of experts increases (tested up to 64), performance continues to climb, whereas monolithic baselines flatline.

3. Efficiency Through Sparsity

An 8-expert ScaleMoE model utilizing a BRC backbone achieved higher returns than a massive monolithic BRC model, while using only 15% of the total active parameters.

4. Functional Specialization and Plasticity

In multi-task settings, the experts naturally specialize. For instance, in HumanoidBench, the gating network learned to route "locomotion" tasks to one set of experts and "manipulation" tasks to another. Because experts only process data they are specialized for, the network maintains its plasticity, significantly reducing the dormant neuron ratio throughout training.

5. State-of-the-Art Results

  • Massive Gains: On the humanoid-run task, scaling to 8 experts yielded a 60% improvement in returns over the single-expert baseline.
  • MetaWorld SOTA: ScaleMoE with 16+ experts significantly outperformed the previous state-of-the-art (a monolithic BRC with a width of 4096) across the 50-task suite.
  • HumanoidBench: With 16 experts, ScaleMoE achieved a normalized return of 0.62—a 27% relative improvement over the monolithic baseline.

Limitations and Future Directions

While ScaleMoE marks a significant step forward, the authors acknowledge a few remaining hurdles:

  • Hyperparameter Complexity: MoE introduces new variables, such as the total number of experts ($N$), the number of active experts ($K$), and the specific gating architecture. Tuning these requires additional effort.
  • Proprioceptive Focus: The current study focuses on low-dimensional state observations (e.g., joint angles, velocities). Scaling MoE for pixel-based RL remains an open challenge that may necessitate novel gating designs.
  • Hardware Overhead: While MoE is highly parameter-efficient, the current implementation in JAX/XLA does not yet fully realize the computational speedups possible with sparse indexing, though frameworks like PyTorch may offer better immediate support.

The Bottom Line: ScaleMoE provides a much-needed scaling axis for deep reinforcement learning. It strongly suggests that the path to truly capable, general-purpose continuous control agents lies in modular, expert-based architectures rather than simply inflating monolithic "black box" networks.

Summary updated

Abstract

Scaling network remains a bottleneck in deep reinforcement learning (RL): simply enlarging actor–critic networks destabilizes training and soon saturates performance. Although recent monolithic architectures such as SimBa and BRC have shown that carefully designed inductive biases can enable positive scaling up to a certain size, their improvements plateau soon as model parameters grow further. This work introduces ScaleMoE, a scalable RL architecture that integrates Mixture-of-Experts (MoE) modules into both the actor and critic of modern continuous control algorithms. Two complementary gating schemes are studied: output-level aggregation of per-expert policies and Q-functions, and feature-level fusion of expert representations before a shared head. We instantiate ScaleMoE on two representative monolithic RL baselines: the single-task method SimBa and the multi-task method BRC. Experiments across the DeepMind Control Suite, MetaWorld, and HumanoidBench show that progressively increasing the number of experts (up to 64) yields substantial improvements in returns, significantly outperforming monolithic networks of comparable or even greater parameter counts. Results demonstrate that ScaleMoE provides an efficient and effective scaling axis for deep RL in continuous control.
BibTeX
@inproceedings{ma2026scalemoe,
  author = {Yi Ma and Chenjun Xiao and Hongyao Tang and Yaodong Yang and Jinyi Liu and Jing Liang and Jiye Liang},
  title = {{ScaleMoE: Mixture-of-Experts for Scalable Continuous Control in Actor-Critic Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=oVS7o5jKJF},
}
Figure