In machine learning, we often want a model to "match" a target distribution—whether that target is a set of expert demonstrations, a pre-trained language model, or a complex physical posterior. The standard tool for this is the Kullback-Leibler (KL) divergence, but the direction of the divergence fundamentally changes what the model learns.
While standard Reinforcement Learning (RL) and Variational Inference (VI) typically minimize the Reverse KL ($D_{KL}(Q || P)$), a growing body of research suggests that the Forward KL ($D_{KL}(P || Q)$) is essential for maintaining diversity and preventing the "mode collapse" that plagues modern alignment and imitation methods.
The Geometry of Mode Covering
The core intuition lies in how these objectives handle regions where the target distribution $P$ has mass.
- Reverse KL is mode-seeking: If the model $Q$ assigns probability to a region where $P$ is zero, the penalty is infinite. Consequently, $Q$ tends to "shrink" to fit inside a single high-probability mode of $P$, ignoring other valid modes to avoid the risk of covering empty space.
- Forward KL is mode-covering: As noted by Wu et al., the Forward KL (or "inclusive" KL) forces the model $Q$ to place mass wherever the target $P$ has mass. If $P(x) > 0$ but $Q(x) \approx 0$, the penalty becomes very large. This "zero-avoiding" property ensures the model captures the full breadth of the target distribution.
In complex tasks with multi-modal latent spaces, Wu et al. argue that this mass-covering property is essential for robust performance. Similarly, in cosmological reconstruction, Modi et al. demonstrate that Forward KL prevents the model from ignoring low-probability but statistically significant tails of the distribution.
Preserving Diversity in LLM Alignment
The "mode-seeking" nature of Reverse KL is a primary culprit for the repetitive, overly cautious behavior seen in aligned Large Language Models (LLMs). When models are fine-tuned using Reinforcement Learning from Human Feedback (RLHF), they often collapse onto a few high-reward responses.
To counter this, Pang et al. propose using Forward KL to encourage models to cover the full distribution of high-quality responses. This perspective is extended to the token level by Zeng et al., who show that token-specific Forward KL constraints preserve the diversity of the original pre-trained model while still steering it toward human preferences.
This regularization is particularly vital for reasoning. Huang et al. identify a "training bottleneck" where standard RL prunes low-probability tokens like "wait" or "however" that are critical for triggering new reasoning paths. By applying Forward KL regularization, the authors shield these "Reasoning Sparks" from being extinguished, allowing for more stable long-term scaling.
Imitation and the Marginal Distribution
In Imitation Learning (IL), the choice of divergence explains the performance gap between simple Behavior Cloning (BC) and complex Inverse Reinforcement Learning (IRL). Ghasemipour et al. and Ghasemipour et al. demonstrate that IRL methods succeed because they implicitly match the expert's state-marginal distribution using a mode-covering objective.
However, mode-covering is not a silver bullet. Ghasemipour et al. warn that Forward KL can cause an agent to broaden its policy excessively, potentially covering suboptimal or conflicting expert trajectories, which might lower expected rewards during evaluation. Despite this, the exploratory benefits are significant; Ramos et al. and Ramos et al. show that Forward KL regularization encourages agents to maintain a broad distribution of physically feasible actions, preventing premature convergence in the early stages of training.
Beyond Mode Collapse in Generative Design
The need for diversity extends to scientific domains like biomolecular design. When fine-tuning diffusion models to generate proteins or small molecules, standard RL can lead to a single "optimal" structure that may be difficult to synthesize.
Lee et al. and Su et al. both argue for Forward KL objectives in these contexts. By treating high-reward samples as a target distribution to be covered rather than a single point to be maximized, these methods ensure the model retains the structural diversity and naturalness of the original data distribution. As Li et al. summarize in their work on Distribution-Matching Policy Optimization (DMPO), approximating the Forward KL allows models to navigate the entire reward landscape, maintaining a diverse library of high-quality solutions rather than collapsing into a single discovered mode.
Key Foundational Work
Understanding the Relation Between Maximum-Entropy Inverse Reinforcement Learning and Behaviour Cloning (2019) — contributed 3 concept elements to this direction.
Amortized Population Gibbs Samplers with Neural Sufficient Statistics (2020) — contributed 2 concept elements to this direction.
A Divergence Minimization Perspective on Imitation Learning Methods (2019) — contributed 1 concept elements to this direction.