Choosing the neural network that turns raw observations into features for a reinforcement learning (RL) agent matters a great deal. Small changes to that network can swing an agent from failure to strong play, and searching over such networks is expensive: every candidate normally needs a full training run. In supervised learning, this cost has been cut sharply by zero-cost proxies, which are scores computed from an untrained network in a single forward and backward pass that predict how well it will train. Nobody has built such a score for RL encoders. The supervised scores are also measuring the wrong thing. RL encoders rarely fail because gradients cannot flow. They fail because their features collapse during training: many states get squeezed into a few directions, gradients for different states start pointing the same way, and the policy update stops being local.
We propose a cheap test applied to each candidate architecture before any environment training. The test feeds the untrained network a fixed batch of frames gathered by a random policy. It then runs a few dozen gradient steps toward deliberately shifting random targets, which imitate the moving targets of value learning. It records how much feature diversity the network keeps and how strongly per-state gradients line up. The claim is that this "collapse stress test" ranks encoder architectures by eventual RL performance better than supervised zero-cost proxies, parameter count, or equal-cost short training runs. If that holds, filtering candidates with it lets architecture search spend its training budget only on architectures that will not collapse.
Core Claim
Three findings motivate the claim. First, representation failure is a central bottleneck in deep RL. No Representation, No Trust shows that Proximal Policy Optimization (PPO) loses feature rank in 5 of 6 Atari games and 7 of 8 MuJoCo tasks. In that regime, per-state gradients become collinear, so clipping the policy ratio no longer keeps updates local. Second, I4R shows that how spread out features are, measured by the number of significant singular values, tracks performance. Third, SimTPR reports a Spearman correlation of 0.54 between pretrained encoder feature rank and downstream return.
These papers supply our measurements. All of them measure representation health during or after training and on a fixed architecture. We ask whether susceptibility to collapse is already largely determined by architectural wiring and can be read off at initialization. That question extends the observation of Saxe et al. that useful properties such as frequency selectivity come from network topology rather than learned weights.
Prior architecture search for RL does not test this. RL-DARTS searches a weighted mixture of operations (a "supernet") and updates it with the RL loss. It costs about 3× a single training run. Its search is driven only by reward, and it degrades when early exploration fails. LACER uses representation quality, in the form of mutual information, as feedback for a language-model architecture designer, but it computes that feedback through online RL training of each candidate. HPO-RL-Bench is "zero-cost" only because it looks up 200,000 precomputed runs, so it cannot score an architecture outside its depth and width grid. The supervised proxies of Zero-Cost Proxies for Lightweight NAS and EZNAS are truly training-free. They measure gradient flow and activation correlations under a static loss, and they are known to fail at separating the top 10% of models.
Our proposal differs from each. It is training-free with respect to the environment, unlike LACER and RL-DARTS. It works on unseen topologies, unlike HPO-RL-Bench. It targets the specific RL failure mode under non-stationary targets, unlike Synflow, SNIP, or EZNAS-A.
Research Program
The test has three components, all computed on one shared batch of about 2,000 frames per environment collected once by a uniform random policy. The first is the number of significant singular values of the penultimate features at initialization, following I4R. This captures whether the wiring itself merges distinct states. The second is the mean absolute cosine similarity between per-frame gradients of a scalar value head. This is the collinearity quantity that Moalla et al. link to trust-region breakdown, evaluated here at initialization.
The third component is the stress test proper. We attach a linear head and run $T$ (about 100) SGD steps on regression targets that are redrawn every 10 steps, imitating bootstrapped targets that move. We then record two quantities. One is the rank retention ratio $\rho = r_T / r_0$, where $r_t$ is the number of significant singular values after $t$ steps. The other is the growth factor of the pre-activation norm, the quantity that Proximal Feature Optimization penalizes.
We combine the components without learned weights, by averaging each architecture's rank on each component. This mirrors the voting ensemble of Abdelfattah et al. and avoids overfitting a combination to the evaluation set. The full test costs seconds per architecture.
Validating any RL proxy requires ground truth that does not yet exist. The main build is therefore a modest tabular benchmark of trained RL encoders. We will draw roughly 200 architectures from a discretized version of the RL-DARTS "Classic" (convolution and pooling) and "Micro" (activation) cell spaces, plus width variants of IMPALA-CNN. Each will be trained with PPO on four Procgen games using 500 training levels, a fixed step budget, and three seeds. We will log return on training levels and on held-out levels, along with feature rank, gradient collinearity, and pre-activation norms throughout training.
These training curves let us check the mechanism and not just the correlation. If the score works, it should predict which architectures collapse during training, and not merely which ones score well. A smaller second set of about 50 architectures trained on Atari 100k with a value-based agent tests whether the score transfers across learning algorithms and data regimes.
Baselines
The naive reference points are the default IMPALA-CNN with no search, and parameter count used as a proxy. Parameter count matters because Cobbe et al. found that scaling IMPALA-CNN up to 16× improves Procgen consistently, so "bigger is better" may explain much of the ranking. Every correlation we report will therefore also be given within parameter-count bins, and as a partial correlation that controls for parameter count. The test beats this reference only if it adds predictive power beyond size.
The strongest training-free competitors are the supervised proxies Synflow, SNIP, and jacob_cov, together with their voting ensemble from Zero-Cost Proxies for Lightweight NAS, plus EZNAS-A from EZNAS. Their reported supervised numbers are the bar for what a good proxy achieves. The ensemble reaches Spearman 0.82 on NAS-Bench-201, and EZNAS-A reaches Kendall τ 0.65 on NAS-Bench-201 CIFAR-10. We found no reported result on Lacuna for any zero-cost proxy's rank correlation with RL return. All proxies will be computed on the same random-policy frames and the same initializations. Supervised proxies that need labels will receive the same random regression targets that our test uses.
We count as beating them a statistically higher Spearman correlation with held-out-level return across seeds, by paired bootstrap over architectures. We also count higher recall of the true top-10% architectures. That second criterion targets exactly the regime where EZNAS reports supervised proxies fail. The SimTPR value of 0.54, which comes from rank measured after pretraining, serves as a reference ceiling-like point for a training-free rank measure.
The strongest search competitor is RL-DARTS, which reports discovered architectures reaching up to 250% of IMPALA-CNN performance on specific Procgen tasks for about 3× the cost of one training run. We will rerun it in our search space on the same games and give our pipeline the same total GPU hours. Our pipeline scores 10,000 candidates with the test and fully trains the top few. We beat RL-DARTS if our selected architecture's held-out return is at least equal at matched compute, or equal at lower compute. Random search under the same budget serves as the floor.
Two ablations isolate the key ingredient. The first uses only the initialization-time rank and collinearity terms, dropping the shifting-target steps. The second runs the stress steps with fixed targets instead of shifting ones. If shifting targets add no predictive power, then non-stationarity is not what the test is capturing, and the RL-specific framing is unjustified.
Why not just…?
Why not just train each candidate briefly and rank by early return? This is the most serious threat. Budgeted estimators such as BPE formalize how short training can preserve rankings. In RL, early return is noisy and policy-dependent, and RL-DARTS documents how poor early exploration misleads search. Still, we will not argue the point; we will measure it. We will run short PPO runs at 0.5%, 2%, and 5% of the full budget and plot rank correlation against compute for both methods. If short training dominates the stress test at every cost level, the claim fails. We would then report the stress test only as a pre-filter in front of short training, and only if that combination still helps.
Why not just use Synflow or another supervised proxy? These proxies are cheap and transfer across vision, language, and speech. However, they measure gradient flow under a fixed loss, while the failures documented by Moalla et al. arise from moving targets. The direct head-to-head comparison and the shifting-versus-fixed-target ablation decide this. If Synflow matches us, the conclusion is that RL encoder ranking is no harder than supervised ranking. That would be a useful finding in its own right.
Why not just fix a standard backbone and add a representation regularizer like I4R or PFO? Regularizers repair collapse on fixed networks: I4R raised DQN median human-normalized score from 15% to 38% across 30 games. If the regularizer makes architectural differences irrelevant, architecture search for collapse resistance is pointless. We will therefore retrain the top, middle, and bottom deciles of architectures, ranked by our test, with PFO added. If PFO erases the gap between deciles, the test's value narrows to two uses: choosing architectures that need no tuned regularizer, which matters because I4R is sensitive to its weighting coefficient, or explaining which architectures need it. We will report whichever outcome occurs.
Why not just scale IMPALA-CNN? Scaling works on Procgen, but it multiplies inference cost. Collapse also occurs in standard deep networks. The parameter-controlled correlations and a compute-matched comparison against 4× and 16× IMPALA-CNN answer this directly.
Decisive Checks
The first check costs no training. For the MLP and CNN depth and width configurations in HPO-RL-Bench, we compute the stress-test score and correlate it with the precomputed PPO and A2C returns. That benchmark already found architectural hyperparameters to matter more than the discount factor. If the score shows no correlation even there, the project stops before any Procgen training.
The main check is the Procgen benchmark of about 200 architectures. The test must achieve a higher parameter-controlled Spearman correlation with held-out return than the best supervised proxy. It must also beat equal-cost short training. Its rank-retention term must predict the feature-rank collapse later observed during PPO training.
The practical check is the matched-compute search against RL-DARTS and random search. The Atari 100k transfer set tests whether a score calibrated on PPO in Procgen still ranks architectures under a different algorithm. Failure on that set would mean that collapse susceptibility depends on the algorithm and not only on the architecture, which would itself be a useful constraint for future RL architecture search.
References
- No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
- I4R: Promoting Deep Reinforcement Learning by the Indicator for Expressive Representations
- On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learning
- On Random Weights and Unsupervised Feature Learning
- Differentiable Architecture Search for Reinforcement Learning
- LLM-Driven Composite Neural Architecture Search for Multi-Source RL State Encoding
- HPO-RL-Bench: A Zero-Cost Benchmark for HPO in Reinforcement Learning
- Zero-Cost Proxies for Lightweight NAS
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Rethinking Performance Estimation in Neural Architecture Search