Learning Energy Decompositions for Partial Inference in GFlowNets

Hyosoon Jang, Minsu Kim, Sungsoo Ahn

TL;DR

LED-GFN (Learning Energy Decompositions for GFlowNets) introduces a novel way to train generative models by "breaking down" a final reward into a sequence of learned intermediate credits. By decomposing terminal energy into smooth, learnable potential functions, LED-GFN solves the credit assignment problem in complex sampling tasks without requiring expensive or noisy intermediate energy evaluations.

The Credit Assignment Problem in GFlowNets

Generative Flow Networks (GFlowNets) are powerful tools for sampling diverse, high-reward objects—such as novel molecules or protein sequences. They construct these objects step-by-step through a sequence of actions. However, they suffer from a classic "credit assignment" problem: if the model only receives a reward (or energy score) after a long sequence of actions, it struggles to identify which specific steps were responsible for the final outcome.

A previous solution, Forward-Looking GFlowNet (FL-GFN), attempted to fix this by evaluating the energy of intermediate, incomplete states. Unfortunately, this approach has two major flaws:

  1. Feasibility: In many real-world tasks, such as drug discovery, an "incomplete" molecule might not have a meaningful energy score, or calculating it might be computationally prohibitive.
  2. Fluctuation: Intermediate energies can be highly misleading. A "bad" intermediate state might eventually lead to a "great" final object, creating noisy training signals that confuse the model.

The LED-GFN Approach

To resolve these issues, the authors propose LED-GFN, which moves away from predefined intermediate energies and instead learns how to decompose the final energy.

Diagram illustrating how LED-GFN decomposes terminal object energy into learnable intermediate potentials along a trajectory.
  • Energy Decomposition: LED-GFN represents the terminal energy $E(x)$ as a sum of learnable "potentials" $\phi_\theta$ associated with each state transition. Essentially, the model learns to distribute the final "bill" across every step of the journey.
  • Variance Regularization: To prevent the model from dumping all the credit onto a single step (which would recreate the sparse reward problem), the authors use a squared loss function with dropout-based regularization. This forces the potential function to change smoothly over the sequence of actions, providing a steady "breadcrumb trail" of dense information for the GFlowNet to follow.
  • Reparameterization: These learned potentials are then plugged into standard GFlowNet objectives (like Detailed Balance or Sub-trajectory Balance), allowing the policy to learn from "local credits" rather than just the final outcome.

Crucially, the paper provides a theoretical guarantee: even if the learned potential isn't a perfect decomposition, the GFlowNet can still be trained to sample from the exact correct Boltzmann distribution using a correction term.

Experimental Validation and Key Results

The authors evaluated LED-GFN across five diverse domains to test its robustness: Bag Generation (a toy task), Set Generation, Molecular Discovery, RNA Sequence Generation, and Maximum Independent Set (MIS) optimization.

Robustness to Noisy Signals

In the "Bag Generation" task, intermediate energies are totally uninformative until the bag is full. FL-GFN fails to improve in this environment because it relies on those uninformative intermediate states. LED-GFN, however, successfully learns the task by creating its own smooth credit assignment.

In the Bag Generation task, LED-GFN learns a smooth accumulation of credits over time, avoiding the sparse reward trap that hinders FL-GFN.

Superiority in Practical Domains

In complex tasks like Molecule and RNA generation, LED-GFN significantly outperformed standard GFlowNets and FL-GFN in both the quality (top-100 scores) and diversity (number of modes) of the samples found. It also demonstrated a lower "relative mean error," meaning it does a better job of matching the true underlying target distribution than its predecessors.

Computational Efficiency

Despite training an extra potential model, LED-GFN only added about 10% computational overhead compared to a standard GFlowNet. In contrast, FL-GFN was nearly twice as slow because it had to call the external energy function at every single step.

Limitations and Future Work

While highly effective, LED-GFN does introduce some additional complexity:

  • Extra Model Complexity: It requires maintaining and training an additional potential function ($\phi_\theta$), which adds a layer of hyperparameter tuning (e.g., the dropout rate $\gamma$).
  • Smoothness Assumption: The method relies on the intuition that a "smooth" credit distribution is optimal. While this held true for the tested tasks, there may be edge cases in combinatorial optimization where rewards are inherently "spiky" and hard to decompose linearly.

Future directions include exploring more sophisticated architectures for the potential function and theoretically analyzing what constitutes an "optimal" local credit for GFlowNet convergence.

Summary updated

Abstract

This paper studies generative flow networks (GFlowNets) to sample objects from the Boltzmann energy distribution via a sequence of actions. In particular, we focus on improving GFlowNet with partial inference: training flow functions with the evaluation of the intermediate states or transitions. To this end, the recently developed forward-looking GFlowNet reparameterizes the flow functions based on evaluating the energy of intermediate states. However, such an evaluation of intermediate energies may (i) be too expensive or impossible to evaluate and (ii) even provide misleading training signals under large energy fluctuations along the sequence of actions. To resolve this issue, we propose learning energy decompositions for GFlowNets (LED-GFN). Our main idea is to (i) decompose the energy of an object into learnable potential functions defined on state transitions and (ii) reparameterize the flow functions using the potential functions. In particular, to produce informative local credits, we propose to regularize the potential to change smoothly over the sequence of actions. It is also noteworthy that training GFlowNet with our learned potential can preserve the optimal policy. We empirically verify the superiority of LED-GFN in five problems including the generation of unstructured and maximum independent sets, molecular graphs, and RNA sequences.
BibTeX
@inproceedings{jang2024learning,
  author = {Hyosoon Jang and Minsu Kim and Sungsoo Ahn},
  title = {{Learning Energy Decompositions for Partial Inference in GFlowNets}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=P15CHILQlg},
}
Figure