Reinforcement learning agents often lock onto the first reward they find. Once a policy favours a mediocre strategy, it stops collecting the experience that would reveal a better one, and the preference becomes permanent. The usual fixes add randomness at the level of individual actions, through entropy bonuses or softened action choice. In deceptive tasks this kind of randomness is the wrong shape. Escaping a dead-end corridor or swimming the "wrong" way toward a distant reward takes many consecutive steps of committed behaviour, and per-step dithering rarely produces that.
Brains face a similar problem when choosing between competing behaviours, and theoretical neuroscience describes a specific solution. Behaviours compete through mutual inhibition, so one wins outright: a "winner-take-all" circuit. The winning population slowly tires as it stays active, which lowers the barrier holding it in place. Random fluctuations then occasionally kick the circuit over into a rival behaviour. We propose putting exactly this circuit on top of a multi-head reinforcement learning agent. Each head is a full policy, and the circuit decides which head drives behaviour, for how long, and when to switch. The resulting exploration comes in long, coherent bouts. Switching is frequent when the heads' estimated values are close and becomes exponentially rare as one head clearly pulls ahead, so exploration anneals itself without an entropy temperature schedule.
Core Claim
A small stochastic competition circuit with fatigue, choosing among $K$ policy heads, should escape deceptive local optima more reliably than per-step entropy methods and more cheaply than diversity archives. The reasons should be measurable. First, it produces temporally extended commitment: dwell times of tens to hundreds of steps per head. Second, its switching rate follows escape-over-a-barrier statistics, so the mean dwell time grows roughly exponentially in the value gap divided by the noise variance. Third, fatigue and noise play separable roles. Fatigue guarantees that a head occupied for a long time is eventually abandoned. Noise makes abandonment probabilistic and sensitive to the value gap.
The third claim speaks to a disagreement in the literature. Attractor Dynamics with Synaptic Depression shows that strong synaptic depression alone can destabilize a held state in a continuous attractor network. Neural model for multi-stability in visual action recognition finds that fatigue only flattens the barrier and noise must supply the kick. Fluctuation-Driven Neural Dynamics Reproduce Drosophila Locomotor Patterns reproduces heavy-tailed fly walking and resting bouts with noise and state-space geometry alone. An RL agent whose performance depends on the switching regime is a clean place to ask which combination is useful, not just which one fits data.
Mechanism
The agent has a shared network trunk and $K$ heads. Each head outputs a policy and an estimate $V_k(s)$ of the return it expects from the current state $s$. A selection circuit holds an activity $x_k$ and a fatigue variable $a_k$ for each head, updated once per environment step:
$$\tau \dot{x}_k = -x_k + f\Big(V_k(s) - \beta a_k - w\sum_{j} x_j\Big) + \sigma \xi_k, \qquad \tau_a \dot{a}_k = -a_k + x_k .$$
Here $f$ is a steep sigmoid. $w$ is the strength of a single shared inhibitory pool, which keeps wiring linear in $K$, following the global-inhibition design described in Modeling the Olfactory Bulb - Coupled Nonlinear Oscillators. $\beta$ sets how strongly fatigue erodes a head's drive, and $\tau_a \gg \tau$ is the slow fatigue timescale. $\xi_k$ is unit Gaussian noise, and the head with the largest $x_k$ acts. We parameterize the noise as a finite-population fluctuation, $\sigma = c/\sqrt{N}$ with $N$ the notional number of units per head population, following the mesoscopic theory of Bridging spiking neuron models and mesoscopic population models - a general theory for neural population dynamics. Population size $N$ is then the single interpretable knob that replaces an entropy temperature.
Two design choices avoid known pitfalls. The first concerns settling time. Dynamically-Adaptive Winner-Take-All Networks needs about 200 cycles to settle on a winner. We never wait for the circuit to settle. It integrates continuously at one step per environment step, and the persistence across steps is the point.
The second concerns diversity among heads. Each head is trained mainly on transitions from bouts it controlled, plus shared replay down-weighted by a mixing coefficient. If all heads trained on all the data, they would collapse onto the same solution, and switching would be meaningless.
Relation to Prior Work
Three papers supply the motivation and ingredients but stop short of this design.
Biologically Plausible Variational Policy Gradient with Spiking Recurrent Winner-Take-All Networks uses winner-take-all clusters as a policy and shows their steady-state rates perform variational inference. It is tested only on InvertedPendulum and reward-based MNIST, and it relies on noise-free steady states, so it has no defence against premature commitment.
Diversity Through Exclusion (DTE): Niche Identification for Reinforcement Learning through Value-Decomposition is the closest existing work. It also uses multiple heads to avoid deceptive niches, but it does so through a learning-rule change. Each head is trained as if other heads' rewards were zero, which permanently segregates the heads, and it has no dynamics governing which head acts when. We change the selection dynamics instead, and we test DTE's exclusion rule as a complementary add-on.
The fatigue-plus-noise mechanism itself comes from Neural model for multi-stability in visual action recognition and Attractor Dynamics with Synaptic Depression. Neither involves reward or learning.
Several papers constrain the design. Resmax: An Alternative Soft-Greedy Operator for Reinforcement Learning shows that exponential, softmax-like selection over-concentrates probability on early favourites. Our circuit's steep nonlinearity risks the same failure, which is why the value-gap and dwell-time analysis below is central rather than decorative. Diversity policy gradient for sample efficient quality-diversity optimization shows that SAC and TD3 fail entirely on its maze tasks. It also shows that quality-diversity methods depend on handcrafted behaviour descriptors, which we do not use.
Research Program
The work has three stages.
The first stage is analytical and synthetic. In a frozen setting with fixed $V_k$, we measure dwell-time distributions as functions of the value gap, $\sigma$, $\beta$ and $\tau_a$. We map out the regimes: noise-only gives exponential dwells, fatigue-only gives regular deterministic alternation, and the combined regime gives peaked but stochastic dwells. We check where the escape-rate prediction holds, namely that log mean dwell is linear in gap over $\sigma^2$, and where the fatigue term breaks it.
The second stage is tabular and gridworld learning. We use RiverSwim and HardSquare from the Resmax paper and the DTE mushroom maze, with Q-learning heads, where the full sweep over $K$, $N$ and $\tau_a$ is cheap.
The third stage uses continuous deceptive control: the MuJoCo swimming snake and quadruped two-passage maze from Reinforcement Learning with Deep Energy-Based Policies, and Ant-Trap from the quality-diversity paper. Here the heads are actor-critic heads on a shared trunk.
Baselines
The naive reference is a single-head agent with standard per-step exploration: epsilon-greedy or softmax in tabular tasks, and DDPG or SAC with a tuned temperature in continuous ones. It shows how badly the deceptive tasks punish per-step dithering. All methods share the same network trunk and environment step budget, and each gets an equal hyperparameter search budget of the same number of configurations over its own exploration knobs.
The strongest single-agent existing method on the continuous deceptive tasks is soft Q-learning from Reinforcement Learning with Deep Energy-Based Policies. On the swimming snake it reports that 100% of soft Q-learning runs found the forward-swimming optimum, while DDPG committed to the backward mode 20% of the time. On the quadruped maze it reports that 100% of soft Q-learning policies found the unblocked passage, versus DDPG succeeding only 60% of the time. Because soft Q-learning already saturates success rate on these two tasks, beating it there means matching 100% success at fewer environment steps and lower wall-clock cost, since soft Q-learning needs amortized Stein variational sampling. Ant-Trap is the task where success rate can genuinely separate methods. On the related AntMaze, QDPG reached high performance in about 8.4×10^7 steps and SAC and TD3 failed entirely, per Diversity policy gradient for sample efficient quality-diversity optimization. We run QDPG with its published descriptors as an upper reference that uses privileged information. Matching it without descriptors would count as a clear win.
In tabular tasks the strongest competitor is Resmax, but there is no reported numerical result on Lacuna for Resmax on RiverSwim or HardSquare; the file states only that it found the optimal policy where softmax and epsilon-greedy often failed. We therefore rerun it ourselves. On the mushroom maze, DTE is the direct competitor, but no reported numerical result on Lacuna was found for DTE beyond its qualitative description of heads partitioning to reach the red mushroom. We rerun it with the same number of heads.
Three ablations isolate the key ingredient. Noise-only sets $\beta=0$. Fatigue-only sets $\sigma=0$. A matched-statistics control replaces the circuit with a random head switcher that samples dwell durations from the circuit's empirical dwell-time histogram, ignoring values. The third control tests whether the value-gap sensitivity of switching matters or only the bout length.
Why not just…?
Why not just resample a softmax over head values every $k$ steps? This is the most serious threat. It gives coherent bouts and value-sensitive head choice with two knobs: temperature and bout length. Its switching is clock-driven rather than state- and history-driven. It cannot leave a head early when that head's value collapses mid-bout, and it cannot abandon a long-occupied head that still looks best, which is the situation in Ant-Trap. We include it with temperature and $k$ tuned on the same budget. If it matches the circuit on all deceptive tasks, we will report that the dynamical mechanism adds nothing beyond bout structure.
Why not just add a count-based or UCB bonus over heads? Fatigue is, functionally, a leaky visitation count, so a head-level UCB rule with a decaying count is a near relative. The differences are that fatigue recovers, so the penalty is forgetful, and that noise makes leaving probabilistic and gap-dependent rather than deterministic. Our fatigue-only ablation is close to a deterministic leaky-count scheduler, and we also run head-level UCB directly. If either equals the full circuit, the contribution reduces to "use a leaky count," and we will say so.
Why not just use DTE's exclusion rule or a stronger entropy bonus? Exclusion changes what heads learn, not how behaviour moves between them. The two can be combined, and we test the combination. Per-step entropy is the wrong temporal shape for tasks that require long commitment, as the reported DDPG and SAC failures on the snake, quadruped maze and AntMaze tasks suggest. The naive baselines quantify how much of that holds under fair tuning.
Decisive Checks
Three results would establish or refute the claim.
First, in learned agents, log mean dwell time per head should be approximately linear in the running value gap between the active head and the runner-up, with slope scaling as $1/\sigma^2$, or equivalently with $N$. If dwells are insensitive to the gap, the escape interpretation is wrong.
Second, on Ant-Trap and the mushroom maze, the full circuit should beat the matched-statistics random switcher and the noise-only ablation in success rate at a fixed step budget. A value-blind switcher with the same bout lengths doing as well would show that bout length, not competition dynamics, is what matters.
Third, the circuit should beat the clock-driven softmax switcher specifically on tasks where the best-looking head must be abandoned, Ant-Trap being the main case. It should not beat that switcher on tasks without such traps, which is a prediction the account can fail.
We also report sensitivity to the single knob $N$ alongside sensitivity of SAC to its temperature. The hope that population size is a more forgiving knob than an entropy temperature is a hypothesis to be tested, not a premise.
References
- Attractor Dynamics with Synaptic Depression
- Neural model for multi-stability in visual action recognition
- Fluctuation-Driven Neural Dynamics Reproduce Drosophila Locomotor Patterns
- Modeling the Olfactory Bulb - Coupled Nonlinear Oscillators
- Bridging spiking neuron models and mesoscopic population models - a general theory for neural population dynamics
- Dynamically-Adaptive Winner-Take-All Networks
- Biologically Plausible Variational Policy Gradient with Spiking Recurrent Winner-Take-All Networks
- Diversity Through Exclusion (DTE): Niche Identification for Reinforcement Learning through Value-Decomposition
- Resmax: An Alternative Soft-Greedy Operator for Reinforcement Learning
- Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
- Reinforcement Learning with Deep Energy-Based Policies