In standard reinforcement learning, we typically search for the "optimal" policy—the single best sequence of actions to maximize a reward. However, this focus on a single point of failure makes agents brittle. If a robot trained to walk on flat ground encounters a slippery surface or suffers a broken motor, its one-and-only "optimal" strategy often fails completely.
A growing body of research suggests a different goal: instead of finding one perfect solution, agents should discover a behavioral repertoire—a library of diverse, high-performing strategies. By maintaining a collection of qualitatively different ways to solve the same task, agents gain a form of functional redundancy that allows for near-instant adaptation to unforeseen changes.
The Optimality Trap and the Need for Diversity
The core problem with traditional RL is "mode collapse" toward a single behavior. When an agent finds a high-reward path, the optimization process reinforces that path until all other alternatives are discarded. Takayuki Osa and colleagues argue in Discovering Diverse Solutions in Deep Reinforcement Learning that this limits an agent's ability to adapt. If the environment changes, a "one-trick pony" has no fallback.
To solve this, researchers use Quality-Diversity (QD) optimization. The goal is to fill a "map" of behavioral niches with the best possible version of each behavior. For example, in Training Diverse High-Dimensional Controllers, Bryon Tjanaka and Matthew Christopher Fontaine demonstrate that a hexapod robot can learn thousands of different gaits. If one leg breaks, the robot doesn't need to relearn how to walk from scratch; it simply identifies which gait in its pre-existing library still works under the new physical constraint.
Maintaining Near-Optimality
A key challenge is ensuring that diversity doesn't come at the cost of performance. We don't just want different behaviors; we want different useful behaviors.
Tom Zahavy and the team at DeepMind addressed this with DOMiNO (Discovering Policies with DOMiNO). They frame diversity as a constraint-satisfaction problem: find the most diverse set of behaviors possible, provided they all stay within a certain threshold of the optimal reward. Similarly, in Discovering Diverse Nearly Optimal Policies with Successor Features, Zahavy and Andre Barreto use Successor Features to represent long-term behavior, allowing the agent to mathematically distinguish between strategies that look similar in the short term but lead to different outcomes.
Generative Models for Behavior
How do we represent and sample from an infinite variety of solutions? Recent work has moved away from discrete "slots" in a library toward continuous generative spaces.
- Diffusion Models: Zechu Li and Pulkit Agrawal introduced DDiffPG (Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient), which uses diffusion to learn multiple "modes" of solving a task. This allows an agent to switch between modes to navigate around unexpected obstacles.
- GANs: Marija Jegorova and Stéphane Doncieux proposed Generative Adversarial Policy Networks (GPNs) (2019 ICDL-EpiRob) to "hallucinate" novel movement variations. If a standard reaching movement is blocked, the robot can sample its internal GAN to find a different trajectory that reaches the same goal.
Zero-Shot and Few-Shot Adaptation
The ultimate "why" of this research is resilience. When an agent possesses a diverse repertoire, adaptation becomes a search problem rather than a learning problem.
In Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity, Xingrui Yu and colleagues show that agents using Extrinsic Behavioral Curiosity (EBC) can achieve zero-shot adaptation. Because they have already pre-discovered a wide variety of "behavioral niches," they can maintain performance during hardware failures (like motor damage) simply by switching to a specialized policy that remains functional.
This redundancy is equally critical in safety-critical domains. Ugo Lecerf and Christelle Yemdji-Tchassi demonstrate in Automatically Learning Fallback Strategies that training diverse policies concurrently provides "fallback" modes for autonomous driving. If a high-risk scenario makes the primary strategy unsafe, the system can instantly pivot to a pre-validated conservative alternative.
By shifting the focus from "the best" policy to "the best set" of policies, this research direction provides a path toward AI systems that are not just efficient in simulation, but robust and survivable in the messy, changing reality of the physical world.