Best Case Reward Maximization in Policy Search

26 concepts20 papers1 proposal

Standard reinforcement learning (RL) is built on a democratic principle: every trajectory matters. By maximizing the expected return, an agent seeks a policy that performs well on average. However, in many high-stakes discovery tasks—such as finding a specific mathematical formula to describe a physical law or a unique sequence of code to solve a logic puzzle—the "average" performance is irrelevant. In these domains, we are looking for a needle in a haystack; we do not care if the agent fails a million times, provided it succeeds once.

Recent research has shifted toward risk-seeking policy optimization, a framework that discards the average in favor of the extreme. By prioritizing the best-case performance over the mean, these methods allow agents to discover rare, high-reward solutions that standard RL would ignore.

The Signal Dilution Problem

In environments with sparse rewards or vast search spaces, the gradient signal in standard RL is often "diluted." If an agent discovers a brilliant solution only 0.1% of the time, the other 99.9% of mediocre or failing attempts will dominate the expected value, pulling the policy back toward "safe" but suboptimal behaviors.

Chris J. Maddison et al. address this by introducing Particle Value Functions, which utilize an exponential utility function to weight trajectory outcomes. This risk-seeking posture expands the "basin of attraction" for optimal paths. Instead of the gradient being near zero in difficult regions, the exponential weighting amplifies the signal from rare successes, making them visible to the optimizer.

This shift is particularly crucial for Large Language Models (LLMs). Christian Walder et al. and Deep Tejas Karkhanis argue that training models to be "right on the first try" (pass@1) forces a conservative bias. By optimizing for a pass@k objective, the model becomes more risk-tolerant, accepting frequent failures in exchange for a higher probability of a single creative breakthrough.

From Average Performance to Symbolic Discovery

The most successful application of risk-seeking gradients has been in Symbolic Regression and discrete search. When searching for a mathematical expression, a "nearly correct" formula is often useless; the goal is the exact symbolic truth.

Brenden K. Petersen et al. proposed a risk-seeking policy gradient (RSPG) that focuses updates entirely on the top-performing fraction of generated candidates. This approach aligns the optimization objective with the nature of the task: identifying a single global optimum. This mechanism has been extended to various "discovery" domains:

  • Physics: Mengge Du et al. use risk-seeking objectives to identify concise partial differential equations, preventing the agent from getting trapped in local optima within sparse search spaces.
  • Control Theory: Haohan Zou et al. apply RSPG to discover analytical Lyapunov functions for nonlinear systems, where the agent must find at least one valid solution that satisfies strict formal constraints.
  • Heuristics: Feng Wu et al. and Hong Wang et al. utilize risk-seeking RL to evolve compact symbolic expressions for branching policies and linear solvers, outperforming traditional "black-box" neural networks.

Escaping Local Optima through Risk

Why does gambling on high-variance outcomes work better than seeking steady progress? The answer lies in the geometry of the loss landscape. Standard Bayesian inference and RL often fail when models are misspecified or when the policy is stuck in a local optimum established during pre-training.

Futoshi Futami et al. demonstrate that risk-seeking optimization prioritizes parameter regions where the loss exhibits high variability. By "gambling" on these high-performing configurations, the optimizer can avoid safe but mediocre local optima. Similarly, Yuhua Jiang et al. show that risk-seeking objectives allow LLMs to traverse low-probability regions of the policy space, effectively breaking the exploration dilemma that plagues standard fine-tuning.

Interestingly, this "risk-seeking" behavior may already be latent in successful algorithms. David Nass et al. discovered that Relative Entropy Policy Search (REPS), a popular information-theoretic method, is mathematically equivalent to a specific case of risk-seeking optimization. This suggests that the inherent "optimism" of these algorithms—their tendency to prioritize high-reward regions—is a primary driver of their success in complex environments.

Summary of Intuition

In a standard RL setting, the agent is a hedger, trying to raise the floor of its performance. In a risk-seeking setting, the agent is a venture capitalist, willing to accept many total losses for the sake of one "unicorn" success. By shifting the objective from the mean to the tail of the distribution, researchers have unlocked the ability for neural networks to solve discrete, symbolic, and sparse-reward problems that were previously considered unreachable.

Key Foundational Work

Particle Value Functions (2017) — contributed 2 concept elements to this direction.
Entropic Risk Measure in Policy Search (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_risk_seeking_policy_gradient_b5413a9f,
  title = {risk-seeking policy gradient},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/risk-seeking-policy-gradient/rdr_b5413a9f194749af97b29863d04cb849},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_risk_seeking_policy_gradient_b5413a9f_v1,
  title = {risk-seeking policy gradient},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/risk-seeking-policy-gradient/rdr_b5413a9f194749af97b29863d04cb849/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake