PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore

Zhihao Lin, Lin Wu, Zhen Tian, Jianglin Lan

TL;DR

PrefPoE is a novel reinforcement learning framework that replaces blind "spray-and-pray" exploration with targeted, advantage-guided action sampling. By fusing a standard policy with an advantage-weighted preference network using a Product-of-Experts (PoE) mechanism, it creates an organic "soft trust region" that prevents entropy collapse and dramatically boosts sample efficiency across both continuous and discrete tasks.

What Problem Does This Paper Address?

In reinforcement learning (RL), balancing exploration (trying new things) and exploitation (acting on known rewards) is a classic challenge. On-policy algorithms like Proximal Policy Optimization (PPO) typically rely on uniform action noise or basic entropy bonuses to encourage exploration.

The authors refer to this as a "spray-and-pray" strategy: it treats all regions of the action space with equal importance. In high-dimensional spaces, randomly searching everywhere becomes incredibly inefficient. Furthermore, as rewards begin to dominate the learning process, standard PPO often suffers from "entropy collapse"—where the policy prematurely converges to a suboptimal behavior because its exploration mechanism fizzles out too early. PrefPoE addresses this by shifting the paradigm from exploration everywhere to exploration where it matters most.

Methods

The authors introduce PrefPoE (Preference-Product-of-Experts), a modular, plug-and-play extension for policy gradient methods. It features a two-branch architecture built on top of a shared backbone encoder:

  1. Main Policy Head: Focuses on standard return maximization (optimizing the classic PPO clipped surrogate objective).
  2. Preference Policy Head: Learns to identify promising regions of the action space. It is trained using a novel advantage-guided objective that balances advantage-weighted likelihood with entropy regularization, mathematically converging to a Boltzmann distribution over advantage estimates.
  3. Product-of-Experts (PoE) Fusion: Instead of linearly averaging the outputs of these two heads (which dilutes confidence), PrefPoE multiplies the distributions ($\pi_{\text{fused}} \propto \pi_\theta \cdot \pi_{\text{pref}}$).

For continuous control tasks modeled with Gaussian distributions, this multiplicative fusion results in a closed-form precision-weighted combination. Crucially, the total variance of the fused distribution is guaranteed to be lower than either individual head. This variance reduction acts as an organic, mathematically emergent "soft trust region" that stabilizes policy updates without requiring hard KL-divergence constraints.

Figure 1: Architecture of PrefPoE. A shared backbone (a common encoder fenc(s)) feeds into main and preference policy heads, which are fused via PoE to generate actions. The pref- erence head is trained with advantage guidance to focus exploration on high-value regions.

Datasets

The framework was evaluated across six diverse, standard simulation benchmarks from OpenAI Gym and MuJoCo:

  • Continuous Control Environments: HalfCheetah-v4 (6D action space), Ant-v4 (8D action space), and LunarLanderContinuous-v2 (2D action space).
  • Discrete Control Environments: CartPole-v1, LunarLander-v2, and FrozenLake-v1 (a challenging, sparse-reward navigation task).

No new datasets or benchmarks were introduced; the authors relied on these widely accepted baselines to demonstrate the generalizability of their approach.

Key Insights and Ideas

  • Quality Over Quantity in Exploration: The paper challenges the conventional RL wisdom that higher entropy always yields better exploration. PrefPoE proves that structured, lower-entropy exploration targeted at high-advantage regions can drastically outperform unstructured, high-entropy random sampling.
  • U-Shaped Entropy Dynamics: Unlike standard PPO where exploration entropy decays monotonically until collapse, PrefPoE exhibits an adaptive, U-shaped entropy curve. The preference network narrows its focus early on to exploit high-advantage actions, but broadens later to maintain diversity as new advantage patterns emerge.
  • Feedback Loop of Guidance: By biasing action sampling toward positive advantages, the samples collected carry a stronger learning signal. This improves the signal-to-noise ratio of the policy gradient, accelerating value function convergence, which in turn provides even better advantage estimates to guide the preference network.
Figure 2: Entropy dynamics in HalfCheetah-v4 demonstrating PrefPoE’s resistance to entropy col- lapse. (a) Preference entropy shows initial focusing and later broadening. (b) PPO entropy decays monotonically, converging near 1.0. (c) PoE entropy shows adaptive U-shaped dynamics, recover- ing and stabilizing around 1.0 (1–2M), thereby preventing the collapse observed in vanilla PPO.

Key Results

PrefPoE demonstrated massive performance leaps and enhanced training stability compared to vanilla PPO and linear fusion baselines:

  • Continuous Action Spaces: PrefPoE achieved a staggering +321% improvement on HalfCheetah-v4 (shattering the baseline score of 1276 to reach 5375), a +69% improvement on Ant-v4, and a +47% gain on LunarLanderContinuous-v2.
  • Discrete Action Spaces: The domain-agnostic architecture delivered a +276% improvement on LunarLander-v2, +68.6% on CartPole-v1, and +30% on FrozenLake-v1.
  • Training Stability: PrefPoE significantly smoothed out variance across random seeds. For instance, on Ant-v4, it slashed the coefficient of variation (CV) from a highly volatile 51.2% (vanilla PPO) down to a stable 13.4%.

Limitations and Future Work

  • Dependence on Critic Quality: Like all advantage-based methods, PrefPoE's exploration guidance is only as good as its advantage estimates. If the value function/critic is poorly calibrated or deceptive early in training, the preference network could focus exploration on suboptimal regions.
  • Consensus Trade-off: The Product-of-Experts mechanism inherently promotes distributional consensus. This creates a strict trade-off between stability and aggressive exploration that might require careful hyperparameter tuning in highly deceptive reward landscapes.
  • Future Directions: The authors highlight several promising avenues, including designing adaptive fusion schedules for different training phases, extending the framework to multi-modal preference networks for complex robotic manipulation, and applying it to Reinforcement Learning from Human Feedback (RLHF).

Summary updated

Abstract

Exploration in reinforcement learning remains a critical challenge, as naive entropy maximization often results in high variance and inefficient policy updates. We introduce \textbf{PrefPoE}, a novel \textit{Preference-Product-of-Experts} framework that performs intelligent, advantage-guided exploration via the first principled application of product-of-experts (PoE) fusion for single-task exploration-exploitation balancing. By training a preference network to concentrate probability mass on high-advantage actions and fusing it with the main policy through PoE, PrefPoE creates a \textbf{soft trust region} that stabilizes policy updates while maintaining targeted exploration. Across diverse control tasks spanning both continuous and discrete action spaces, PrefPoE demonstrates consistent improvements: +321\% on HalfCheetah-v4 (1276~$\rightarrow$~5375), +69\% on Ant-v4, +276\% on LunarLander-v2, with consistently enhanced training stability and sample efficiency. Unlike standard PPO, which suffers from entropy collapse, PrefPoE sustains adaptive exploration through its unique dynamics, thereby preventing premature convergence and enabling superior performance. Our results establish that learning \textit{where to explore} through advantage-guided preferences is as crucial as learning how to act, offering a general framework for enhancing policy gradient methods across the full spectrum of reinforcement learning domains. Code and pretrained models are available in supplementary materials.
BibTeX
@misc{lin2025prefpoe,
  author = {Zhihao Lin and Lin Wu and Zhen Tian and Jianglin Lan},
  title = {{PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore}},
  year = {2025},
  eprint = {2511.08241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.08241},
}
Figure