Discovering Diverse Policies Through Constrained Optimization

27 concepts17 papers1 proposal

Standard reinforcement learning (RL) algorithms are designed to find a single optimal solution. By maximizing a scalar reward function, an agent converges to a narrow, highly specialized behavior. However, this "one-trick pony" nature makes RL agents brittle. If the environment changes slightly at test time—such as a robot encountering a new obstacle or a sudden shift in friction—the single optimal policy often fails completely.

To build robust, adaptable agents, we instead want to discover a repertoire of behaviorally diverse, high-performing policies. The naive way to achieve this is through linear scalarization: adding a diversity or novelty bonus directly to the task reward ($R_{\text{total}} = R_{\text{task}} + \lambda R_{\text{diversity}}$).

But linear scalarization suffers from a fundamental flaw: gradient conflict. If the diversity weight $\lambda$ is too high, the agent prioritizes novelty over task success, resulting in bizarre, useless behaviors. If $\lambda$ is too low, the diversity incentive is swallowed by the primary reward, and the population collapses back into a single strategy.

To resolve this tension, a powerful paradigm has emerged: Performance-Constrained Diversity Optimization. Instead of treating diversity as a competing reward component, this research direction frames the search for variety as a constrained optimization problem. By enforcing performance boundaries, agents can safely explore the full geometry of the solution space without sacrificing utility.


The Core Intuition: Decoupling Quality from Variety

The core insight of this field is that task performance and behavioral diversity should not be traded off linearly. Instead, we should maximize diversity subject to a minimum performance threshold, or maximize rewards subject to a minimum diversity constraint.

Mathematically, rather than unconstrained scalar maximization, the problem is framed as a Constrained Markov Decision Process (CMDP):

$$\max_{\pi} \text{Diversity}(\pi) \quad \text{subject to} \quad J(\pi) \ge (1-\epsilon)J^*$$

where $J(\pi)$ is the expected task reward of policy $\pi$, $J^*$ is the optimal reward, and $\epsilon$ defines the tolerance for near-optimality.

As Sun et al. demonstrate in Novel Policy Seeking with Constrained Optimization, treating novelty as a hard constraint completely bypasses the gradient interference that plagues multi-objective RL. This allows the agent to remain focused on the primary task while forcing it to explore entirely distinct regions of the policy space.

Kumar et al. pioneered this perspective with SMERL (One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL), showing that applying diversity rewards only to policies that already achieve a near-optimal reward threshold prevents the emergence of irrelevant or failing behaviors. This constraint ensures that every discovered skill in the agent's repertoire is fully functional.


Mechanism 1: Dynamic Balancing via Lagrange Multipliers

The most common mechanism for enforcing these boundaries is the use of learned Lagrange multipliers. Instead of manually tuning a fixed diversity weight $\lambda$, the optimization algorithm adaptively scales the importance of diversity based on whether the performance constraint is currently satisfied.

  Is Task Performance >= Threshold?
        /                  \
     [Yes]                 [No]
      /                      \
Prioritize Diversity     Prioritize Task Reward
(Increase Lambda)        (Decrease Lambda)

Zahavy et al. formalized this in Discovering Diverse Nearly Optimal Policies withSuccessor Features (DSP) by leveraging Successor Features to represent long-term behavior under a CMDP framework. They later extended this concept to state-action occupancy spaces with DOMiNO (Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality), proving that a constrained formulation reliably maps out the true Pareto front between quality and variety where linear scalarization fails.

This Lagrangian balancing acts as an automated governor across various implementations:


Mechanism 2: Phased, Orthogonal, and Stratified Optimization

Beyond explicit Lagrangian dual variables, researchers have designed geometric and structural mechanisms to isolate diversity exploration from performance degradation.

Phased and Switching Optimization

Rather than optimizing both objectives simultaneously, some frameworks decouple them across time. For example, Chen et al. introduced DGPO (DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization), which alternates between maximizing rewards and maximizing diversity once a target reward threshold is achieved.

Similarly, the single-author work by Zhou (Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization) utilizes an adaptive "reward-switching" mechanism that filters trajectories, restricting task-based optimization exclusively to novel behaviors to prevent the agent from collapsing back into previously discovered high-reward paths.

Orthogonal Gradient Projections

In deep learning models, an infinite number of parameter configurations can yield the same optimal loss. Gong et al. exploited this redundancy via Population Gradient Descent (How to Fill the Optimum Set? Population Gradient Descent with Harmless Diversity). Their framework treats diversity as a secondary objective, constraining diversity-seeking updates to be strictly orthogonal to the primary loss gradient. This ensures the population explores varied solutions entirely within the optimal subspace without harming primary task performance.

Performance Stratification

When optimizing a population for diversity, high-performing agents can easily look behaviorally distinct from low-performing agents for trivial reasons (e.g., simply moving faster). To combat this, Wu et al. introduced Quality-Similar Diversity (Quality-Similar Diversity via Population Based Reinforcement Learning). By stratifying diversity optimization into distinct performance intervals, they ensure agents only compete for novelty against their skill peers (e.g., "diverse beginners" vs. "diverse masters"), preventing the optimization process from being dominated by trivial differences.

Performance Tier 3 (Masters)   --> [Agent A] <-> Diversity Search <-> [Agent B]
Performance Tier 2 (Adepts)    --> [Agent C] <-> Diversity Search <-> [Agent D]
Performance Tier 1 (Beginners) --> [Agent E] <-> Diversity Search <-> [Agent F]

Why It Works: Real-World Robustness and Natural Behavior

Constraining diversity optimization yields profound practical benefits over standard RL, primarily because it mirrors the multi-faceted nature of real-world tasks.

Out-of-Distribution Generalization

By maintaining a soft lower bound on performance during exploration, frameworks like DUPLEX by León et al. (Discovering Creative Behaviors through DUPLEX: Diverse Universal Features for Policy Exploration) map out a robust Pareto frontier. When faced with out-of-distribution environmental shifts, the agent can instantly switch to an alternative, pre-discovered strategy that remains functional under the new conditions.

This is highly valuable in complex multi-agent setups. For instance, Shiroshita et al. combined performance filtering with diversity rewards to simulate realistic human traffic, capturing the unpredictable variety of human driving without allowing agents to exhibit failing behaviors. In fine-tuning scenarios, Hamid et al. demonstrated in Polychromic Objectives for Reinforcement Learning that a joint performance-diversity objective prevents models from collapsing into a single narrow strategy, preserving the rich behavioral variety acquired during pretraining.

Eliminating Unnatural Behaviors (Reward Hacking)

Standard unconstrained RL agents often achieve high scores by exploiting simulators—exhibiting jittery, non-naturalistic movements like constant spinning or shaking.

Ho et al. inverted the constrained diversity paradigm to solve this in Towards Human-Like RL: Taming Non-Naturalistic Behavior in Deep RL via Adaptive Behavioral Costs in 3D Games. By enforcing competitive task performance as a primary constraint while minimizing behavioral costs as a secondary objective, their framework ensures that behavioral refinement and smoothing only occur within the subspace of highly competent policies. The result is an agent that wins games while moving with the natural fluidity of a human player.

Key Foundational Work

One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL (2020) — contributed 2 concept elements to this direction.
Discovering Diverse Nearly Optimal Policies withSuccessor Features (2021) — contributed 2 concept elements to this direction.
Novel Policy Seeking with Constrained Optimization (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_constrained_diversity_optimization_6279d2d9,
  title = {constrained diversity optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constrained-diversity-optimization/rdr_6279d2d93ffa462180dfe8d4e7ffcec1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_constrained_diversity_optimization_6279d2d9_v1,
  title = {constrained diversity optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constrained-diversity-optimization/rdr_6279d2d93ffa462180dfe8d4e7ffcec1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake