In many complex search spaces—from the vast chemical combinations of drug discovery to the intricate reasoning paths of large language models—finding a single "best" solution is often insufficient. Standard reinforcement learning (RL) and optimization techniques are designed to maximize expected rewards, a goal that frequently leads to "mode collapse," where the model converges on a single, repetitive optimum.
Reward-proportional sampling offers a different objective: instead of finding the maximum, the goal is to sample candidates with a probability exactly proportional to their reward. This approach, primarily realized through Generative Flow Networks (GFlowNets), ensures that the model discovers and maintains a diverse library of high-quality solutions.
The Problem of Greedy Collapse
Traditional RL agents are typically "greedy"; they seek the path of highest return. As noted by Emmanuel Bengio et al., this causes models to collapse to a single mode, ignoring other regions of the state space that might be nearly as good. In scientific discovery, this is a critical failure. Because reward functions (like molecular docking scores) are often imperfect proxies for real-world performance, relying on a single "optimal" candidate is risky.
Moksh Jain et al. argue that a representative set of candidates is essential for navigating the uncertainty inherent in drug and material discovery. By sampling proportionally to reward, GFlowNets act as a safeguard, providing a diverse pool of candidates that are more likely to contain a truly viable solution when moved to physical validation.
Modeling Generation as a Flow
The core intuition behind GFlowNets is to treat the construction of a discrete object (like a molecule or a graph) as a flow through a network. Imagine a Directed Acyclic Graph (DAG) where the source is an empty state and the sinks are completed objects. Each edge represents an action that adds a component to the object.
To achieve reward-proportional sampling, the model must satisfy a flow consistency principle. As explained by Emmanuel Bengio et al., the total flow entering a state must equal the total flow leaving it, and the flow out of a terminal state must equal its reward. When these conditions are met, the probability of following a specific trajectory is naturally proportional to the reward at its end.
This framework allows for more efficient credit assignment than traditional methods. Jain et al. introduced Trajectory Balance (TB), an objective that enforces consistency across entire generation paths simultaneously. This global view accelerates training and improves the diversity of the generated samples compared to local learning rules.
Diversity as a Mechanism for Robustness
The ability to capture multiple modes is not just about variety; it is a mechanism for robustness. In combinatorial optimization, such as vehicle routing or scheduling, a single "optimal" plan might fail due to real-world noise or hardware latencies.
David W. Zhang et al. demonstrated that by sampling a diverse set of high-quality schedules, systems become more resilient to the inaccuracies of proxy models. Similarly, in the Kidney Exchange Problem, Golnoosh Farnadi et al. showed that generating a pool of near-optimal transplant plans improves individual fairness and provides alternatives when logistical failures occur.
Aligning and Diversifying Language Models
The principles of reward-proportional sampling are increasingly applied to Large Language Models (LLMs) to prevent the generation of repetitive or "cookie-cutter" responses. Standard alignment techniques like DPO often overfit to a single type of preferred response.
To counter this, Oh Joon Kwon et al. proposed GDPO, which treats alignment as a sampling problem. By learning to generate outputs proportional to human preference rewards, the model maintains creativity and diversity while remaining highly aligned. This logic extends to multi-step reasoning; Fangxu Yu et al. found that training models to sample reasoning trajectories proportional to their rewards prevents mode collapse and encourages the discovery of multiple valid ways to solve a problem.
Balancing Exploration and Exploitation
While diversity is the primary goal, practitioners often need to control the "greediness" of the sampler. In large search spaces, allocating too much probability to low-reward regions can be inefficient.
Elaine Lau et al. introduced QGFN, which incorporates action-value functions ($Q$) into the GFlowNet framework. This allows for a tunable parameter that can "dial up" the focus on high-reward regions at inference time without losing the underlying structural diversity. Similarly, Marco Jiralerspong et al. explored interpolating between reward accumulation and reward dilution, helping samplers balance the discovery of new modes with the need to exploit the highest-performing candidates.
Key Foundational Work
Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation (2021) — contributed 2 concept elements to this direction.
Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation (2021) — contributed 2 concept elements to this direction.
Bayesian Structure Learning with Generative Flow Networks (2022) — contributed 2 concept elements to this direction.