Abstract
This paper studies generative flow networks (GFlowNets) to sample objects from the Boltzmann energy distribution via a sequence of actions. In particular, we focus on improving GFlowNet with partial inference: training flow functions with the evaluation of the intermediate states or transitions. To this end, the recently developed forward-looking GFlowNet reparameterizes the flow functions based on evaluating the energy of intermediate states. However, such an evaluation of intermediate energies may (i) be too expensive or impossible to evaluate and (ii) even provide misleading training signals under large energy fluctuations along the sequence of actions. To resolve this issue, we propose learning energy decompositions for GFlowNets (LED-GFN). Our main idea is to (i) decompose the energy of an object into learnable potential functions defined on state transitions and (ii) reparameterize the flow functions using the potential functions. In particular, to produce informative local credits, we propose to regularize the potential to change smoothly over the sequence of actions. It is also noteworthy that training GFlowNet with our learned potential can preserve the optimal policy. We empirically verify the superiority of LED-GFN in five problems including the generation of unstructured and maximum independent sets, molecular graphs, and RNA sequences.
Introduction: The Credit Assignment Problem in GFlowNets
Generative Flow Networks (GFlowNets) are a powerful class of models designed to sample diverse objects (like graphs or sequences) with probabilities proportional to a given reward or "energy" function. However, they face a major hurdle: credit assignment.
In many tasks, a reward is only obtained once the object is completely finished. For instance, in building a molecule atom-by-atom, intermediate "half-finished" states often lack a meaningful score. This makes it difficult for the model to determine which specific early actions led to a high-quality final result. Previous attempts, like Forward-Looking GFlowNets (FL-GFN), tried to calculate the energy of intermediate states, but this is often computationally expensive or physically impossible for incomplete objects. Furthermore, intermediate energies can fluctuate wildly, providing misleading signals that steer the model away from optimal trajectories.
Methodology: Learning Energy Decompositions (LED-GFN)
The authors propose LED-GFN, which moves away from calculating intermediate energies and instead learns how to decompose the final terminal energy into local credits.
The core of the LED-GFN approach involves:
- Energy Decomposition: Modeling the terminal energy as a sum of learnable "potential functions" assigned to each transition in the generation sequence.
- Dense Local Credits: Using a squared loss with dropout-based regularization to force the model to distribute credit smoothly across the entire trajectory. This creates a "dense" training signal that is easier for the policy to follow.
- Reparameterization: These learned potentials are plugged into GFlowNet training objectives (like Detailed Balance or Sub-trajectory Balance), acting as learned "internal rewards."
- Online Learning: The potential function is trained alongside the GFlowNet policy using the trajectories the model currently explores.
Key Results and Performance
LED-GFN was evaluated across five distinct domains, including molecule discovery and RNA sequence generation. The results demonstrate that learning the "why" behind a reward leads to significantly better exploration and discovery.
Key findings include:
- Superior Mode Discovery: LED-GFN found significantly more unique high-reward solutions (modes) than standard GFNs or FL-GFN, as shown in the bag generation task.
- Real-World Performance: In molecule and RNA generation, LED-GFN consistently achieved higher "top-100" average scores and discovered more diverse high-reward samples.
- Efficiency: By avoiding the bottleneck of calling expensive energy oracles for every intermediate state, LED-GFN makes the training process more efficient while maintaining theoretical convergence guarantees.
Conclusion
LED-GFN addresses the sparse reward problem in GFlowNets by learning to decompose terminal energy into informative local credits. By regularizing these potentials to change smoothly, the model provides a stable and dense training signal that enhances the discovery of diverse, high-quality samples. Future work may explore more complex architectures for the potential functions and extend this framework to broader reinforcement learning settings where rewards are sparse.
Summary updated