Grounded Fitness for Critic Populations in Noisy Observation Spaces

Some reinforcement learning methods train a whole population of value estimators (critics, the networks that predict how much future reward an action will bring) and keep the best ones. To avoid the cost of testing each candidate in the environment, the cheapest of these methods score each critic by how well it agrees with its own one-step bootstrapped targets on stored data. That score is the temporal-difference (TD) error. We argue that this score rewards the wrong thing, and that the failure is worst when observations contain many irrelevant, noisy dimensions. A critic can lower its TD error in two ways. It can memorize stored transitions through spurious noise features, or it can inflate its values in a way that stays consistent with themselves. Neither makes it more accurate. Selecting critics by TD error can therefore actively breed overestimation, which is the tendency of learned values to drift upward because the max over noisy estimates picks up positive errors. It also collapses the population onto one brittle representation.

The fix we propose is cheap. Score each critic against returns that actually happened, using fresh trajectories that the elite agents collect anyway. Selection then favors critics whose predictions match reality, at no extra environment cost. The new element is the use of this grounded score as the population's selection signal. The test is whether that choice keeps population-based critics from overestimating and from converging prematurely once observations are padded with hundreds to thousands of distractor dimensions.

Core Claim

Value-Evolutionary-Based Reinforcement Learning (VEB-RL) evolves a population of Q-networks and ranks them by negative TD error on replay minibatches. This makes evolution "sample-free", meaning no rollouts are spent on evaluation, and it improves DQN, SPR and TD3. The research file itself notes the weakness of this score. A degenerate critic can have near-zero TD error while being useless. More generally, TD error measures self-consistency, not accuracy.

Our claim has three parts. First, under high-dimensional observation noise, TD-error ranking systematically promotes critics that overestimate and that put weight on distractor coordinates. Second, it drives the representation homogenization identified by Reducing Overestimation Bias by Increasing Representation Dissimilarity in Ensemble Based Deep Q-Learning. That paper showed ensemble critics collapsing to identical representations and thereby reintroducing overestimation. Third, replacing the fitness with a bias measured against fresh Monte Carlo returns removes both effects while keeping the population's exploration benefit.

The grounded signal comes from Automating Control of Overestimation Bias for Continuous Reinforcement Learning. That work estimates aggregated bias on a "fresh" buffer of recent trajectories and uses it to tune one scalar knob, such as dropped quantiles in TQC, inside a single agent. We reuse the signal for a different job: deciding which critic in a population survives. A scalar knob can only shift all value estimates up or down. Selection can throw away a critic that has latched onto noise and keep one that has not. That is a difference in kind, and it is what the experiments test.

Concretely, let $\mathcal{F}$ be the fresh buffer of the most recent elite-collected episodes, $G(s,a)$ the observed discounted return from $(s,a)$ onward, and $Q_i$ the $i$-th critic. The fitness is

$$f_i = -\frac{\left|\mathbb{E}_{\mathcal{F}}\left[Q_i(s,a) - G(s,a)\right]\right|}{\mathbb{E}_{\mathcal{F}}\left[|G(s,a)|\right]} - \beta\,\sqrt{\mathrm{Var}_{\mathcal{F}}\left[Q_i(s,a) - G(s,a)\right]}.$$

The first term is the normalized signed bias, in the same form as Kuznetsov and Grishin's controller. The second term penalizes state-dependent error spread, with weight $\beta$. Because the second term needs only forward passes on a few thousand fresh pairs, it partially addresses that paper's stated open problem that per-state bias is too expensive to isolate.

Research Program

The system is a VEB-style population of $M$ TD3 critics with a shared actor and a shared replay buffer. Only elites interact with the environment, as in VEB-RL. We change exactly one thing, the fitness function, and keep the GA mutation, the periodic injection of the gradient-trained critic, and the elite interaction protocol.

We will construct observation noise of known structure, so that "attending to noise" can be measured directly. The original MuJoCo or MinAtar observation is concatenated with $k$ distractor coordinates, with $k \in \{0, 64, 512, 2048\}$. There are two noise types. The first is i.i.d. Gaussian noise whose per-dimension variance exceeds that of the true state features. Prior work shows that in this setting variance-based projection keeps noise and discards signal (Dimensionality Reduction for Spectral Clustering; Deep Unsupervised Feature Selection by Discarding Nuisance and Correlated Features). The second is a slowly drifting random walk. Its temporal correlation makes it look predictive within a replay batch, so it gives a critic something spurious to memorize.

We will log three diagnostics per critic.

The first is grounded bias against fresh returns, the quantity in the fitness. It is also logged for TD-fitness runs, where it is used only for measurement.

The second is the share of input-gradient norm that falls on distractor coordinates. We call this the "noise attention" of a critic, and it is measurable because we know which coordinates are fake.

The third is population representation diversity. It is measured with the inequality indices of Sheikh and Bölöni and with diffusion spectral entropy from Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy. That estimator damps small-scale ambient noise by powering a diffusion operator, and it stays informative above 10,000 dimensions where classical estimators saturate. We use it only as a measurement tool.

Together these diagnostics let us test the mechanism, not just the final return. The prediction is that TD-fitness elites show rising bias, rising noise attention and falling diversity as $k$ grows, while grounded-fitness elites do not.

On overhead, the fresh buffer is filled by elite rollouts the method already performs, so extra environment steps are zero. Extra compute is $M$ forward passes over $|\mathcal{F}|$ pairs per generation, which is smaller than VEB-RL's own TD evaluation over replay minibatches. We will report GPU-hours alongside returns throughout, since the linear cost of populations is the standing objection to this whole family (Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks).

Baselines

The naive reference point is single-agent TD3 from Addressing Function Approximation Error in Actor-Critic Methods. It uses clipped double-Q, delayed policy updates and target smoothing, and runs on the same noisy observations with its published hyperparameters. The file reports TD3 reaching an average return of about 9636 on HalfCheetah-v1. That figure anchors the $k=0$ setting, and a correct reimplementation should approach it. To make the comparison compute-fair, we also run TD3 with $M$ critics and min-over-critics targets. This matches the population's parameter count and gradient updates, so that any gain cannot be credited to sheer critic count. On MinAtar the analogous references are DQN and MaxminDQN.

The strongest single-agent competitor is AdaTQC from Automating Control of Overestimation Bias for Continuous Reinforcement Learning. It uses the same fresh-buffer bias signal we rely on, so it isolates whether a population adds anything beyond on-the-fly bias tuning. The file reports that it matches or exceeds grid-searched scores on MuJoCo and PyBullet, but no reported numerical result was found on Lacuna for AdaTQC. We will run it with its released defaults and the same environment-step budget.

The strongest population competitor is VEB-RL with TD3 and TD-error fitness. It is identical to our method except for the fitness, which makes it the cleanest comparison. No reported numerical result was found on Lacuna for VEB-RL on MuJoCo. The file reports roughly 100% improvement over DQN on MinAtar Asterix and Seaquest and a 40% gain over SPR on Atari 100k. We will run it from the same code path, with the same $M$, generation schedule and mutation operators. Each method gets the same hyperparameter-tuning budget at $k=0$, after which settings are frozen across noise levels. This prevents us from quietly retuning our method for noise.

For context on population methods that pay for rollouts, we include DvD-TD3 from Effective Diversity in Population Based Reinforcement Learning. It reports a median reward over 6,000 on Humanoid within 1 million steps. We also include CERL from Collaborative Evolutionary Reinforcement Learning, which reports about 4700 on Humanoid within 1 million steps where its individual TD3 learners failed. Both are run with matched environment steps, and we report their GPU-hours.

Three ablations isolate the key ingredient. The first, held-out TD fitness, uses TD error on a reserved replay split; it separates "overfitting to noise" from "self-consistent inflation". The second sets $\beta=0$, testing whether the spread term matters beyond aggregate bias. The third is a monitoring-only variant: it computes the grounded bias but feeds it to an AdaTQC-style truncation knob instead of to selection. This tests directly whether the gain comes from selection rather than from the signal itself.

Beating the field means two things. At $k \ge 512$, grounded-fitness VEB must exceed both TD-fitness VEB and AdaTQC in interquartile-mean (IQM) return over 10 seeds, with bootstrap confidence intervals that do not overlap. It must also show lower grounded bias and lower noise attention. At $k=0$ it must at least tie them.

Why not just…?

Why not just use a single well-tuned TD3 or AdaTQC agent? This is a real threat on easy tasks. CERL itself is less sample-efficient than a tuned TD3 when good hyperparameters are known. The evidence for populations comes from deceptive tasks. Individual TD3 learners failed on Humanoid and Swimmer where CERL succeeded, and DvD-TD3 escaped the "stand still" trap on Humanoid. But none of that evidence involves observation noise. The AdaTQC comparison across $k$ settles the question directly. If AdaTQC matches us at high $k$, then a scalar bias knob suffices, the population is not justified, and we will say so.

Why not just compute TD error on a held-out replay split? Held-out TD error would catch critics that memorize noise, and it is the simplest fix for VEB-RL. Our hypothesis is that it does not catch overestimation that propagates consistently through bootstrapping. A critic that inflates every value by a similar amount still satisfies its own Bellman targets, on held-out data too. The held-out TD ablation tests this. If it closes most of the gap, the contribution shrinks to "use a validation split". We will then report that result honestly, since it is still useful to the population-based RL community.

Why not just project away the noise with PCA before learning? When distractor variance exceeds signal variance, PCA keeps the wrong axes. On noisy UCI Iris, PCA-based clustering reached 51–59% accuracy against 68% for constraint-guided reduction (Knowledge Driven Dimension Reduction for Clustering). PCA plus spectral clustering reached NMI 0.75 against DRSC's 0.87. We include a PCA front end on all methods as an ablation. The random-walk noise condition is designed so that a fixed projection cannot remove it cleanly. If PCA plus TD3 nonetheless matches our method, then the problem is solved by preprocessing, not by selection.

Why not just add diversity regularizers to the critic ensemble? Inequality regularizers counter representation collapse (Reducing Overestimation Bias by Increasing Representation Dissimilarity in Ensemble Based Deep Q-Learning). But they reward being different, and under noise a critic can be different by attending to different noise coordinates. Grounded fitness rewards being right. We test this by adding Gini-regularized critics to the TD-fitness population and checking whether their noise attention rises with $k$.

Proposed Experiments

The main experiment runs Hopper, Walker2d, HalfCheetah, Ant and Humanoid for 1 million steps at each distractor level $k$ and each noise type. The discrete counterpart runs MinAtar Asterix, Breakout, Seaquest and Space Invaders with distractor channels. Both use 10 seeds, $M=5$ critics, IQM return with bootstrap intervals, and GPU-hours reported for every method. No reported results with appended distractor dimensions were found on Lacuna for any baseline, so all noisy-setting numbers will be produced by us under one shared code path.

We have three decisive checks.

The first is the rank test. Within each generation, compute Spearman correlation between each critic's fitness and the return of a policy improved against it for a short fixed budget. The prediction is that TD-error fitness correlates well at $k=0$ but degrades toward zero or negative as $k$ grows, while grounded fitness stays positive. This tests the mechanism without relying on end-to-end return alone.

The second is the bias trajectory test. Over training, TD-fitness elites should show grounded bias climbing with $k$, while grounded-fitness elites stay near zero. This also checks for over-pessimism, the known failure of min-clipping in sparse-reward settings. If our populations drift negative and stall exploration, the absolute value in $f_i$ is not doing its job.

The third is the collapse test. Diffusion spectral entropy and the Gini index over elite critic representations should fall faster under TD fitness than under grounded fitness, and the fall should coincide with rising noise attention.

The proposal is refuted under any of three outcomes: held-out TD fitness matches grounded fitness, AdaTQC matches it at high $k$, or the rank test shows no degradation of TD fitness under noise. Each would point to a simpler method that the field should use instead.