Standard reinforcement learning (RL) is built on a democratic principle: every trial matters. When we train an agent to navigate a maze or play a video game, we typically optimize for the expected return—the average reward across many episodes. This "average-case" optimization encourages agents to be reliable and conservative, avoiding catastrophic failures to keep the mean score high.
However, in scientific discovery, we don't care about being "good on average." If a researcher is searching for a new room-temperature superconductor, a specific mathematical proof, or a high-performance code kernel, they don't need a policy that produces mediocre results consistently. They need a policy that produces a single, world-changing outlier.
This shift from "average-case" to "best-case" optimization is the core of a growing body of research into risk-seeking policy gradients.
The Mismatch of Average Rewards
In tasks like symbolic regression—where the goal is to find a concise mathematical formula that fits a dataset—the reward landscape is incredibly sparse. As noted in Deep Symbolic Optimization: Reinforcement Learning for Symbolic Mathematics, standard RL objectives that maximize average reward are ill-suited for these tasks. If an agent spends 99% of its time generating nonsense equations and 1% of its time finding the exact physical law, an average-case objective will penalize the agent for the 99% failure rate, potentially washing out the signal of the breakthrough.
Mert Yuksekgonul and colleagues argue in Learning to Discover at Test Time that traditional RL focuses on generalization across many tasks, which is poorly suited for "eureka" moments. When the goal is to identify a single high-performing outlier, the optimization target must shift from the mean of the distribution to its extreme upper tail.
From Mean to Max: The Risk-Seeking Objective
To solve this, researchers have introduced the entropic risk measure and risk-seeking policy gradients. Instead of updating the model based on the average performance of all sampled candidates, these algorithms prioritize the "best-of-k" samples.
As David Nass and Jan Peters demonstrate in Entropic Risk Measure in Policy Search, information-theoretic policy search methods can be mathematically framed as risk-seeking optimization. This framework allows an agent to be "optimistic," prioritizing high-variance regions of the search space where the potential for a massive reward exists, even if the average outcome is poor.
This is often implemented using a Pass@K objective. In Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems, the authors explain that by training a model to maximize the probability that at least one of its $K$ attempts is successful, the model becomes more risk-tolerant. It stops trying to be "safe" and starts targeting narrow, high-reward peaks that standard reward-averaging would ignore.
Fishing for Lucky Streaks
The intuition behind this approach is captured by Thomas Keller 0001 and Florian Geißer in their paper Better Be Lucky than Good. They show that in evaluation settings where only the best runs count, a "mixed" strategy that alternates between a safe optimal policy and a high-variance "fishing" policy can outperform a standard optimal policy.
By strategically seeking out "lucky streaks," the agent maximizes its probability of achieving an extreme positive outlier. This "fishing" behavior is exactly what risk-seeking policy gradients aim to automate in high-dimensional search spaces.
Applications in Symbolic and Combinatorial Discovery
This shift in objective has enabled breakthroughs across several domains where "best-case" performance is the only metric that matters:
- Symbolic Regression: Frameworks like DISCOVER and DYSYMNET use risk-seeking gradients to find exact governing equations for physical systems. By focusing on the top-performing fraction of samples, these models can navigate the vast combinatorial space of mathematical operators to find the single "correct" formula.
- Code Optimization: In PerfDojo, Andrei Ivanov and team use this approach to discover high-performance GPU kernels. They target the single fastest execution schedule rather than the average speed, allowing the RL agent to find non-intuitive hardware shortcuts.
- Mathematics and Graph Theory: Researchers at Google DeepMind used similar principles in Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Search, noting that standard RL objectives must be modified to prioritize the discovery of a single extremal instance over average performance.
- Formal Verification: Haohan Zou and colleagues apply risk-seeking objectives in Analytical Lyapunov Function Discovery to find symbolic stability certificates for complex systems. Here, the goal is to find at least one valid mathematical expression that satisfies strict formal constraints.
Why It Works: Breaking the Convergence Trap
The primary benefit of risk-seeking objectives is that they prevent the "collapse" into safe but mediocre solutions. As Chenglei Si and Tatsunori Hashimoto observed in their work on Automated AI Research, standard RL often struggles because it converges on "safe" ideas.
Risk-seeking gradients provide a mathematical "license to explore." By ignoring the low-reward floor and focusing entirely on the high-reward ceiling, these methods allow reinforcement learning to function as a powerful engine for scientific and mathematical discovery, where the only thing that matters is the breakthrough.