Searching for a good neural network design for a reinforcement learning (RL) agent is expensive for two reasons. Each candidate design must be trained, and an RL agent collects its own training data, so a candidate that explores badly early on looks bad for reasons unrelated to its design. We propose to make these two costs pay for each other. A weight-sharing "supernet" is one large network that contains many candidate sub-networks, which reuse the same weights. At the start of every episode the agent samples one sub-network and runs it, without changing it, until the episode ends. Because different sub-networks behave differently but each behaves consistently within an episode, this sampling acts as parameter-space exploration: the agent explores by perturbing its policy's internals rather than jittering each action. Each episode's return is also a free, full-length evaluation of the sub-network that produced it. Exploration and architecture evaluation therefore become the same rollouts.
The approach also addresses a known scaling problem. Perturbing policy weights explores well over long horizons, but its sample cost grows with the number of parameters perturbed (Contrasting Exploration in Parameter and Action Space), and supernets are deliberately overparameterized. Sampling which operations are active is a perturbation in a space of perhaps tens of discrete choices rather than millions of weights. Our bet is that the behavioral diversity this low-dimensional perturbation induces is enough to explore with.
Core Claim
Freezing the sampled sub-network for a whole episode gives temporally coherent exploration whose dimensionality is the size of the architecture space, not the weight count. Updating the architecture distribution on episode returns is then a policy-gradient estimator in architecture space whose variance does not grow with horizon. The resulting search should resist the early-exploration collapse reported for differentiable search in RL. It should do so at roughly the cost of one training run, because no rollouts are spent purely on evaluation. The new ingredient is the episode-level commitment to a discrete sub-network, with sampling temperature adapted to a target behavioral divergence. Simply adding weight noise to a supernet would not include either piece.
Where This Sits
Two works motivate the proposal. Differentiable Architecture Search for Reinforcement Learning (RL-DARTS) puts a continuous softmax mixture of candidate operations inside actor-critic networks. It reports search at about 3× one training run and about 30× the efficiency of random search. It explores only through action noise or entropy bonuses, and its authors identify early exploration failure, which degrades the architecture weights, as a core weakness. Parameter Space Noise for Exploration and Noisy Networks For Exploration show that perturbing weights instead of actions escapes local optima that action noise falls into. We keep the parameter-noise principle but move the perturbation from weights to the discrete architecture choice.
Two works constrain the design. The analysis in Contrasting Exploration in Parameter and Action Space shows that parameter perturbation is horizon-independent but scales with dimension $d$. This rules out naively perturbing all supernet weights. Disturbance-immune Weight Sharing shows that shared-weight rankings can be near random, with Kendall's τ of 0.16, which threatens our claim that episode returns are a useful evaluation signal. We must measure that rank correlation in RL rather than assume it.
The closest existing method is Neural Architecture Evolution in Deep RL for Continuous Control (ACN). ACN runs a population of distinct-architecture TD3 agents that feed one shared replay buffer, so architectural diversity already shapes the data. ACN differs from our method in three ways. It maintains separate networks. It changes architectures through genetic mutation and distillation. Its environment exploration still comes from TD3 action noise. We use one shared-weight network, re-sample architectures every episode, and treat that resampling as the exploration mechanism itself.
Searching for Robust Binary Neural Networks via Bimodal Parameter Perturbation does add noise to both weights and architecture logits. It does so in supervised binary-network search, as a fixed-variance regularizer that flattens the loss landscape, not as exploration. Single Path One-Shot NAS supplies our sampling primitive: train a supernet by activating one random path at a time. SPOS resamples per minibatch in supervised learning. We resample per episode, and the timing of the resampling is the substance of our claim.
Research Program
We build a supernet actor, with an optional shared critic, over a small discrete space. On MuJoCo and DM-Control the space consists of per-layer choices of width, activation, and skip connection in an MLP. On Procgen it is a reduced cell space over the IMPALA-CNN backbone. The space is kept small enough, on the order of 100 to a few hundred configurations, that ground truth is obtainable. A categorical distribution $p_\phi$ over operation choices, with logits $\phi$, governs sampling. At each episode start we draw an architecture $a \sim p_\phi$ and hold it fixed. Supernet weights train with the base RL loss, SAC or TD3 off-policy and PPO on-policy. For PPO, the probability ratios are computed under the same sub-network that generated the episode.
The architecture distribution is updated on standardized episode returns:
$$\phi \leftarrow \phi + \eta\, \mathbb{E}_{a \sim p_\phi}\big[(R(\tau_a) - b)\,\nabla_\phi \log p_\phi(a)\big],$$
Here $R(\tau_a)$ is the return of the episode run with architecture $a$, $b$ is a batch-mean baseline, and $\eta$ is a step size. This is the parameter-based policy gradient of PGPE applied to architecture variables. Like PGPE, it scores whole episodes rather than individual steps, so its variance does not accumulate over the horizon.
The second component borrows the adaptive scaling of Plappert et al. We periodically measure the mean action-space divergence between pairs of sampled sub-networks on replay states. We then adjust a sampling temperature on $p_\phi$ so that this divergence tracks a target $\delta$, which decays over training. This keeps exploration from vanishing when shared weights make sub-networks nearly identical. It also keeps behavior from becoming erratic when sub-networks differ too much.
The analyses ask three questions. First, how much behavioral diversity does architecture sampling produce, compared with weight noise at matched divergence? Second, does the episodic-return ranking of sub-networks inside the supernet correlate with standalone performance? Third, how does total search cost compare with RL-DARTS? Cost is measured in environment steps and GPU-hours, including retraining of the final architecture.
Baselines
The naive reference point is the default fixed architecture trained with standard action noise: IMPALA-CNN with PPO on Procgen, and the standard MLP with SAC or TD3 on MuJoCo. It receives the same environment-step budget as the full search plus retraining. Beating it is necessary but not sufficient.
The strongest existing competitor is RL-DARTS (Miao et al.), run on the same search space with joint optimization of weights and architecture logits as its authors recommend. It is the only method in the file that searches RL architectures at near single-run cost. Its reported results are discovered cells reaching up to 250% of IMPALA-CNN performance on specific Procgen games, and search time of about 3× a standard run. No reported DM-Control or MuJoCo scalar returns for RL-DARTS were found on Lacuna, so we will produce those numbers ourselves. For fairness, both methods receive the same total environment steps, the same backbone, the same base RL algorithm, and the same hyperparameter-tuning budget on a held-out subset of games. Beating RL-DARTS means one of two things. Either we reach higher retrained return on the final architecture at equal steps, or we match it with lower variance across seeds and fewer collapsed searches, where a collapsed search is defined in advance as a final architecture below the default baseline.
The obvious combination, RL-DARTS with NoisyNet layers (Fortunato et al.) in the supernet, gets its own row. Its tuning budget for initial noise scale matches the budget for our target $\delta$. ACN (Franke et al.) is compared on MuJoCo Humanoid, Ant, and HalfCheetah at matched environment steps. There, ACN's [672, 508] Humanoid architecture outperformed the TD3 baseline, which stalled. Random search over standalone-trained architectures, at the same total environment steps, gives the cost reference.
Two ablations isolate the key ingredient. In per-step switching, the same $p_\phi$ is resampled at every timestep rather than every episode, which removes temporal consistency while keeping identical diversity. The second ablation removes adaptive temperature.
Why not just…?
Why not just add NoisyNet or adaptive weight noise to the RL-DARTS supernet? This is the most direct competitor, and it might work. The cost analysis of Vemula et al. predicts that perturbing all supernet weights pays a dimension penalty that architecture sampling avoids. The softmax mixture also still collapses for reasons unrelated to exploration, as NAS-Bench-201 showed for DARTS (Dong and Yang). These are predictions, not evidence. The NoisyNet-supernet row at matched behavioral divergence settles the question directly. If it matches our method, the architecture-subspace claim is wrong, and the contribution reduces to "parameter noise fixes RL-DARTS," which is still worth reporting.
Why not just resample sub-networks every minibatch or every step, as SPOS does? Per-minibatch resampling during learning is compatible with our method and is in fact how weights train. The question is how the behavior policy is chosen during rollouts. The per-step-switching ablation answers it. Plappert et al. and Deep Coherent Exploration show that step-level noise loses the consistency that matters on long horizons. Deep Coherent Exploration also found an intermediate correlation optimal, so we additionally test switching every $k$ steps. If per-step switching matches per-episode on the long-horizon sparse tasks, our central mechanism is falsified.
Why not just tune the entropy bonus or warm up the supernet weights before letting architecture logits move? Warm-up is a real threat, because it plausibly removes early collapse without any exploration change. We include RL-DARTS with a uniform-logit warm-up phase, with its length tuned within the shared budget. On sparse tasks, Plappert et al. report that action noise failed outright on SparseMountainCar and SparseCartpoleSwingup while parameter noise succeeded. Warm-up does not change the action-noise data the supernet sees, so we expect it to fail on those tasks even if it closes the gap on dense ones. If it closes the gap everywhere, we will report that.
Why not just score architectures with zero-cost proxies and skip interaction? Zero-cost proxies score networks at random initialization on fixed supervised batches (Zero-Cost Proxies for Lightweight NAS). Nothing in the file validates them under the shifting data of RL. We test this cheaply by computing such a proxy for every architecture in our small spaces and reporting its rank correlation with standalone return alongside ours.
Proposed Experiments and Decisive Checks
We run on Procgen (all 16 games, PPO), DM-Control (SAC), and Gym MuJoCo (TD3/SAC), plus the sparse variants SparseMountainCar, SparseCartpoleSwingup, and SparseHalfCheetah. On SparseHalfCheetah, prior parameter-noise methods report failure, and we expect undirected architecture noise to fail as well. We will report it as a stated limit.
The headline metric is the retrained final-architecture return at matched total environment steps, with 10 seeds. We report the collapse rate and the GPU-hours from search start to final model.
For ground truth, we train 40 architectures from each small space standalone, with 3 seeds each. We then report Kendall's τ between their standalone returns and their mean in-supernet episode returns. The reference point is τ = 0.16 for plain supervised weight sharing in Niu et al.. No reported RL τ was found on Lacuna.
Three outcomes decide the proposal.
- If per-episode sampling beats per-step switching on long-horizon and sparse tasks, the exploration claim holds; if not, it is falsified.
- If our method beats or matches NoisyNet-RL-DARTS at lower parameter dimension of perturbation, the claim that architecture sampling is the better place to put the noise holds.
- A τ meaningfully above that supervised baseline would support the claim that exploration rollouts double as evaluation.
If exploration helps but τ stays near zero, the method is a good explorer and a poor searcher. We would then add a short standalone fine-tuning stage over the top-$k$ sampled architectures and charge its cost against the budget.
References
- Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective
- Differentiable Architecture Search for Reinforcement Learning
- Parameter Space Noise for Exploration
- Noisy Networks For Exploration
- Disturbance-immune weight sharing for neural architecture search
- Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control
- Searching for Robust Binary Neural Networks via Bimodal Parameter Perturbation
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Efficient Sample Reuse in Policy Gradients with Parameter-Based Exploration
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Deep Coherent Exploration for Continuous Control
- Zero-Cost Proxies for Lightweight NAS