Standard reinforcement learning (RL) is designed to find the single best way to solve a task. While effective, this often results in "one-trick pony" agents that are brittle to environmental changes. If a robot learns only one way to walk and its motor fails, it has no fallback. If a driving agent learns only one way to navigate an intersection, it cannot adapt to sensor noise or unusual traffic patterns.
A growing body of research focuses on discovering a repertoire of diverse, near-optimal policies. The goal is to find multiple solutions that are qualitatively different in behavior but similarly high-performing in reward. The key insight across these works is that diversity should not be a competing reward, but a constrained optimization problem.
The Failure of Diversity as a Reward
Early attempts to encourage variety often added a "diversity bonus" to the reward function. However, Finale Doshi-Velez and Muhammad A. Masood noted in Diversity-Inducing Policy Gradient that this balance is incredibly sensitive. If the diversity weight is too high, the agent prioritizes being "different" over being "good," leading to useless or failing behaviors.
This "gradient conflict" occurs because the objective of maximizing reward often points in a different direction than the objective of maximizing novelty. To solve this, researchers like Hao Sun and Dahua Lin proposed Interior Policy Differentiation (IPD) in Novel Policy Seeking with Constrained Optimization. Instead of mixing the two goals, they treat novelty as a hard constraint. The agent is free to be as diverse as possible, provided it stays within a "near-optimal" region of the policy space.
Framing Diversity as a Constraint
The most successful modern approach to this problem is the Constrained Markov Decision Process (CMDP). In this framework, the agent maximizes diversity subject to a performance threshold.
Tom Zahavy and colleagues at DeepMind formalized this in DOMiNO and Diverse Successive Policies. By treating diversity as a constraint-satisfaction problem, they ensure that every discovered behavior—whether it’s a robot learning five different gaits or a navigator finding three different paths—meets a minimum "optimality ratio."
To manage this trade-off dynamically, these algorithms often use Lagrange multipliers. As explained by Arrasy Rahman and Peter Stone in their work on Minimum Coverage Sets, the Lagrange multiplier acts as a self-adjusting pressure valve. If the agent's performance drops below the threshold, the multiplier reduces the pressure to be diverse, refocusing the agent on the task. Once performance is restored, the pressure to differentiate increases again.
Measuring "Different": From Actions to Successor Features
To encourage diversity, we must first define what makes two policies "different." Simply looking at different actions is often insufficient; two agents might take different steps but end up following the same physical path.
- State-Space Distance: Sunli Chen and Yi Wu introduced SIPO in Iteratively Learn Diverse Strategies with State Distance Information, which measures diversity based on the physical distance between states visited. This allows agents to discover complex, human-interpretable strategies, like different passing plays in a football simulation.
- Successor Features: In Quality-Diversity Actor-Critic, Luca Grillotti and Borja G. León use Successor Features—a representation of the long-term occupancy of states. This allows the agent to distinguish between policies based on their long-term behavioral "footprint" rather than immediate actions.
- Harmless Diversity: Chengyue Gong and Qiang Liu take a geometric approach in Population Gradient Descent. They ensure diversity is "harmless" by constraining diversity-seeking updates to be orthogonal to the gradient of the primary task. This ensures that the search for variety never degrades the quality of the solution.
Why Constrained Diversity Works
The shift from unconstrained exploration to constrained diversity provides three major benefits:
- Robustness to Change: In DUPLEX, Borja G. León demonstrates that agents with a library of diverse, near-optimal policies can adapt to "out-of-distribution" changes—like a sudden change in gravity—by simply switching to a different policy in their repertoire.
- Ad Hoc Teamwork: To cooperate with unknown partners, an agent must be prepared for many different "conventions." Muhammad M. Rahman shows in Minimum Coverage Sets that training against a mathematically grounded, diverse set of partners makes agents significantly more robust to "strangers" than training against a single optimal partner.
- Learning from Experts: Even in offline settings, where an agent cannot interact with the environment, Marin Vlastelica and Georg Martius show in Diverse Offline Imitation that Fenchel duality can be used to extract a wide variety of skills from a static dataset while ensuring they all remain "expert-like."
By treating diversity as a boundary to be explored rather than a reward to be chased, these frameworks allow AI agents to move beyond being "one-trick ponies" toward becoming versatile, multi-skilled systems.