Reward Matching for Diversity Preserving Alignment

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2272 papers378 conceptsfrozen 2026-10-04

In the standard paradigm of Reinforcement Learning from Human Feedback (RLHF), we train models to maximize a reward signal. Whether we are fine-tuning a Large Language Model (LLM) to be more helpful or a diffusion model to be more aesthetic, the objective is typically to find the single "best" output for a given prompt.

However, this pursuit of the global maximum often leads to a phenomenon known as mode collapse. When a model is pushed to maximize expected reward, it tends to "sharpen" its distribution, eventually converging on a narrow set of repetitive, safe, or "boring" responses that it knows will score well. This creates a fundamental tension between alignment (following the reward) and diversity (maintaining creative variety).

A growing body of research suggests a different path: instead of maximizing rewards, we should match them.

From Maximization to Proportionality

The core intuition behind this shift is to treat alignment as a sampling problem rather than an optimization problem. In a reward-matching framework, the goal is to ensure that the probability of generating a specific output $x$ is directly proportional to its reward $R(x)$.

$$P(x) \propto R(x)$$

If one response is twice as good as another, the model should generate it twice as often—but it should still generate the second response occasionally. This approach, pioneered through the mathematical framework of Generative Flow Networks (GFlowNets) by Yoshua Bengio and colleagues in GFlowNet Foundations, allows models to explore the entire landscape of high-quality solutions rather than getting trapped in a single peak.

Why Reward Matching Prevents Collapse

Standard RL algorithms like PPO or the popular Direct Preference Optimization (DPO) often suffer from "distribution sharpening." As Oh Joon Kwon and others argue in GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets, these methods overfit to the reward model's preferences.

By contrast, reward-matching objectives like those in GDPO or FlowRL (Xuekai Zhu et al.) treat the generative process as a flow through a state-space. By enforcing "flow consistency"—ensuring the amount of probability flowing into a state matches the reward at the end of the path—the model naturally maintains the diversity of the original pre-trained distribution while steering it toward higher-reward regions.

This is particularly critical in "one-to-many" tasks. For example, in Enhancing Question Generation through Diversity-Seeking Reinforcement Learning, Tianyu Ren demonstrates that reward matching prevents a model from using the same repetitive formula for every question, instead capturing the inherent variety of natural language.

Scaling to Diffusion and Continuous Spaces

The benefits of diversity-preserving alignment extend beyond text. In the realm of image generation, fine-tuning diffusion models with standard RL often leads to "reward hacking," where the model produces distorted but high-scoring images.

To solve this, Zhen Liu and Dinghuai Zhang introduced $\nabla$-GFlowNet (Nabla-GFlowNet). By incorporating the gradients of the reward function directly into the GFlowNet objective, they allow diffusion models to align with human aesthetic preferences while preserving the "creative prior" of the original model. Similarly, the DAG (Diffusion Alignment with GFlowNets) framework (Zhang and Zhang) frames the denoising process itself as a trajectory in a flow network, achieving a superior trade-off between image quality and sample variety.

The Pareto Frontier of Discovery

In scientific domains like drug discovery or protein design, diversity isn't just a "nice-to-have"—it's a requirement. If a model generates 1,000 molecules that are all minor variations of the same chemical structure, the search has failed, even if that structure has a high predicted reward.

Methods like TacoGFN (Mohit Pandey et al.) and Multi-Objective GFlowNets (Moksh Jain et al.) use reward matching to cover the "Pareto front"—the set of optimal trade-offs between conflicting goals (like potency vs. toxicity). By sampling proportionally to the joint reward, these models maintain a library of structurally distinct candidates, ensuring that if one lead fails in the lab, others are available.

Beyond GFlowNets: Simplified Diversity Mechanisms

While GFlowNets provide the theoretical backbone, other researchers are finding simpler ways to achieve similar "un-sharpening" effects.

  • SimKO (Yi Ren and Weiyang Liu) redistributes rewards across the top-K most likely reasoning paths to prevent the model from concentrating all its probability on a single answer.
  • CD-RLHF (Haoran Sun et al.) adds an "intrinsic curiosity" reward to the alignment process, encouraging the model to explore novel but high-quality outputs.
  • Energy Rank Alignment (ERA) (Shriram Chennakesavalu et al.) treats rewards as physical energy states, decoupling the intensity of the reward from the regularization to prevent the model from collapsing into deterministic structures.

Summary

The shift from reward maximization to reward matching represents a fundamental change in how we think about AI alignment. By moving away from the search for a single "optimal" output and instead training models to represent the full distribution of high-quality solutions, researchers are creating AI that is not only more aligned with human preferences but also more creative, robust, and useful for complex discovery.

Papers in this version

Showing 60 of 272 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 137 papers
  2. Version 1 · 2026-10-04 · 272 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_reward_proportional_sampling_e6e39290,
  title = {reward-proportional sampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-proportional-sampling/rdr_e6e392907ef345ce9feb264380ff538d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_reward_matching_for_diversity_preserving_e6e39290_v1,
  title = {Reward Matching for Diversity Preserving Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-matching-for-diversity-preserving-alignment/rdr_e6e392907ef345ce9feb264380ff538d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}