Standard reinforcement learning (RL) and controller synthesis are designed to find the "best" way to solve a task. By maximizing a reward signal or minimizing a cost function, these algorithms typically converge to a single, narrow, optimal policy. While efficient, this "one-trick pony" approach creates a single point of failure: if the environment changes slightly, a sensor fails, or an obstacle appears in the primary path, the system has no fallback and fails entirely.
A growing body of research suggests that robustness stems not from finding the single best solution, but from discovering a repertoire of diverse, high-quality behaviors. By learning multiple ways to achieve the same goal, agents can adapt to unforeseen constraints at runtime without needing to restart the learning process.
The Fragility of the Single Optimal Path
In traditional synthesis, such as the methods discussed by Klaus Dräger and Marta Kwiatkowska, a controller is often a rigid script. In Permissive Controller Synthesis for Probabilistic Systems, the authors argue that this lack of flexibility makes systems vulnerable. If the unique action selected by the controller becomes impossible—perhaps due to a hardware fault—the system is paralyzed.
This brittleness is equally prevalent in deep RL. As Jubayer Ibn Hamid and Chelsea Finn observe in Polychromic Objectives for Reinforcement Learning, standard fine-tuning often causes models to "collapse" into a single strategy, discarding the rich variety of behaviors they might have seen during pretraining.
Building a Behavioral Repertoire
To solve this, researchers are moving toward "Quality-Diversity" (QD) and "Diversity-Seeking" RL. The goal is to fill a "library" of policies that are all nearly optimal but qualitatively different.
One approach is to use generative models to "hallucinate" different solutions. Marija Jegorova and Stéphane Doncieux introduced Generative Adversarial Policy Networks (GPNs) to model the distribution of successful policies. Instead of learning one path, the robot learns a distribution, allowing it to try a different "version" of a behavior if the first one is blocked Generative Adversarial Policy Networks for Behavioural Repertoire. Similarly, Zechu Li utilizes diffusion models in DDiffPG to maintain multimodal behaviors and avoid mode collapse Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient.
Diversity as a Constraint, Not a Reward
A common pitfall in diversity-seeking RL is "gradient conflict." If you simply add a "novelty reward" to the standard task reward, the agent often struggles to balance the two, leading to behaviors that are either diverse but useless, or useful but repetitive.
To fix this, Tom Zahavy and Yannick Schroecker proposed DOMiNO (Diversity Optimization Maintaining Near Optimality). Instead of a trade-off, they treat diversity as a Constrained Markov Decision Process (CMDP). The agent is told: "Maximize diversity, but only among policies that achieve at least 90% of the optimal reward" Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality.
This framing ensures that every discovered behavior is a viable "fallback strategy," a concept also explored by Ugo Lecerf for safety-critical driving Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios.
Measuring Difference with Successor Features
How do you mathematically define "different" behaviors? Simply looking at the raw actions isn't enough; two different sequences of muscle movements might result in the exact same walking path.
Researchers like André Barreto and Volodymyr Mnih use Successor Features (SF) to solve this. SFs represent the long-term occupancy of a policy—essentially, "where does this policy spend its time?" By maximizing the distance between the SFs of different policies, algorithms like DSP (Diverse Successive Policies) can find behaviors that are qualitatively distinct in their interaction with the environment, such as a robot learning to walk, hop, and crawl to reach the same destination Discovering Diverse Nearly Optimal Policies with Successor Features.
The Theoretical Link to Robustness
Why does diversity actually help? Benjamin Eysenbach and Sergey Levine provided a rigorous answer by proving that Maximum Entropy (MaxEnt) RL is mathematically equivalent to solving a robust RL problem.
By maximizing entropy (a form of diversity), an agent is implicitly preparing for a "budget" of environmental disturbances. A MaxEnt agent doesn't just find the shortest path; it finds the "widest" path—one that remains successful even if the physics of the world are slightly altered Maximum Entropy RL (Provably) Solves Some Robust RL Problems.
Summary of the Shift
The transition from single-policy optimization to diverse-set synthesis represents a fundamental shift in how we define "intelligence" in autonomous systems:
- From Scripts to Menus: Systems like those from Marta Z. Kwiatkowska provide "multi-strategies" or menus of safe actions rather than a single command.
- From Accuracy to Adaptability: Algorithms like DUPLEX by Borja G. León focus on "out-of-distribution" (OOD) performance, using a library of skills to handle shifting gravity or new obstacles Discovering Creative Behaviors through DUPLEX.
- From Conflict to Constraints: Frameworks like IPD (Interior Policy Differentiation) by Hao Sun treat novelty as a constraint to avoid the pitfalls of traditional multi-objective optimization Novel Policy Seeking with Constrained Optimization.
By prioritizing a repertoire of near-optimal behaviors, these researchers are building agents that don't just solve a task once, but possess the strategic depth to solve it in any environment.