Standard reinforcement learning (RL) typically operates under a single-minded objective: find the single policy that maximizes expected reward. While computationally efficient, this paradigm creates a fragile single point of failure. As Tom Zahavy et al. argue in Discovering Diverse Nearly Optimal Policies withSuccessor Features, if an agent's physical configuration changes or the environment shifts, a single-solution approach lacks the necessary redundancy to adapt or recover.
Consider a quadruped robot optimized to run forward using a specific bounding gait. If one of its motors fails, or if it encounters a patch of ice, that single optimal strategy becomes completely unviable. Luca Grillotti et al. emphasize in From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Real-World Unsupervised Quality-Diversity that without a diverse repertoire of alternative behaviors, a robot cannot effectively adapt when its primary movement strategy fails due to mechanical damage.
To solve this, a growing body of research focuses on behavioral redundancy: training agents to discover and maintain a library of qualitatively distinct, near-optimal solutions for a single task.
The Intuition of Behavioral Redundancy
Why does behavioral redundancy work? The core intuition is that sub-optimal training paths can become the primary successful strategies when the environment is perturbed. Instead of treating alternative solutions as noise to be discarded, agents preserve them as a safety net.
Aviral Kumar et al. demonstrated this phenomenon empirically in One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL. They showed that as the magnitude of environmental perturbations increases, the optimal behavior naturally shifts between different policies within a pre-trained diverse repertoire. By providing broader coverage of potential test-time conditions, a set of distinct, near-optimal solutions ensures that at least one strategy remains functional.
This approach shifts the adaptation burden from online retraining to strategic selection. As Christian Daniel et al. note in Hierarchical Relative Entropy Policy Search, maintaining a library of diverse strategies allows for immediate adaptation to environmental changes or hardware failures without requiring extensive retraining.
Mechanisms for Discovering Diversity
How do we force an RL agent to learn multiple ways of solving the same task without collapsing into a single best mode? Researchers employ several distinct mechanisms to balance the trade-off between task performance (quality) and behavioral variety (diversity).
Quality-Diversity and Deep Neuroevolution
One prominent approach scales evolutionary algorithms to discover diverse behavioral niches. Cédric Colas et al. developed ME-ES in Scaling MAP-Elites to Deep Neuroevolution, demonstrating that maintaining a diverse archive of high-performing behaviors allows robots to rapidly recover from physical damage by selecting a functional alternative from a pre-evolved library. To improve sample efficiency, Valentin Macé et al. introduced Diversity policy gradient for sample efficient quality-diversity optimization, which integrates a dedicated diversity gradient directly into deep reinforcement learning frameworks.
Constrained Optimization and Successor Features
Another strategy frames diversity as a formal mathematical constraint. In Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality, Tom Zahavy et al. framed diversity as a constrained optimization problem within the space of state-action occupancies. By enforcing that new policies must visit different states or perform different actions while remaining within a bounded threshold of optimal reward, they construct a robust behavioral basis.
Generative and Diffusion Policies
Rather than maintaining discrete policy archives, agents can model a continuous manifold of behaviors using generative models. Zechu Li et al. developed DDiffPG in Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient, combining diffusion models with mode-specific Q-learning to prevent mode collapse. Similarly, Kenneth Derek proposed Adaptable Agent Populations via a Generative Model of Policies, which maps a low-dimensional latent space to a broad policy space, enabling rapid adaptation by searching the latent space for a functional policy in seconds.
Test-Time Adaptation and Resilience
The ultimate value of behavioral redundancy manifests at test time, particularly when agents encounter out-of-distribution (OOD) environments or adversarial challenges.
When an agent faces unexpected obstacles, a multimodal policy can pivot seamlessly. As Alberta Longhini et al. observe in Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies, policies that maintain multiple behavioral modes exhibit significantly higher robustness to environmental changes and reward shifts because they can pivot to alternative successful strategies when a primary mode is blocked.
This resilience is equally powerful when extracting skills from static datasets. Takayuki Osa et al. show in Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning that maintaining a library of diverse behavioral solutions—including those that are sub-optimal in the training environment—facilitates rapid few-shot adaptation to unforeseen mechanical failures or environmental constraints. Furthermore, Pavel Kolev et al. demonstrate in Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints that maximizing diversity under imitation constraints allows agents to bypass obstacles rather than failing when the primary expert path is blocked.
Ultimately, behavioral redundancy changes the fundamental philosophy of reinforcement learning: instead of finding the perfect solution for a static world, it prepares the agent with a rich toolkit of diverse strategies, ensuring survival in a dynamic and unpredictable one.
Key Foundational Work
Hierarchical Relative Entropy Policy Search (2012) — contributed 2 concept elements to this direction.
Two-Timescale Learning Using Idiotypic Behaviour Mediation For A Navigating Mobile Robot (2010) — contributed 1 concept elements to this direction.