q-exponential family for policy optimization

Lingwei Zhu, Haseeb Shah, Han Wang, Martha White

Introduction

In the realm of continuous action space reinforcement learning (RL), the Gaussian distribution has long served as the default choice for parameterizing policies. Its mathematical tractability and ease of sampling make it a convenient starting point. However, the "thin tails" of the Gaussian distribution—where probability density drops off exponentially—often lead to practical challenges. Specifically, policies can suffer from premature convergence, collapsing into narrow peaks that stifle exploration, and a lack of robustness to noisy reward signals or outliers.

This research explores a more flexible alternative: the $q$-exponential family. By generalizing standard exponential distributions through an entropic index $q$, this family allows for a spectrum of policy shapes, ranging from heavy-tailed distributions that sustain exploration to light-tailed distributions that offer sparse support.

Generalizing the Policy Distribution

The $q$-exponential family introduces a tunable parameter $q$ that fundamentally alters the distribution's tail behavior. When $q=1$, the family recovers the standard Gaussian distribution.

Comparison of Gaussian, q-Gaussian, and Student's t distributions. Heavy-tailed versions (q > 1) maintain higher probability in the tails, while light-tailed versions (q < 1) can have finite support.

As illustrated in Figure 1, varying $q$ provides significant flexibility:

  • Heavy-Tailed ($q > 1$): Distributions like the Student’s t and Cauchy fall into this category. They maintain higher probability density far from the mean, which encourages sustained exploration and provides robustness against outliers.
  • Light-Tailed/Sparse ($q < 1$): These distributions can exhibit finite support, assigning exactly zero probability to actions outside a specific range. This property is potentially valuable for safety-critical applications or for mimicking sparse target policies.

Technical Implementation

Integrating the $q$-exponential family into modern RL frameworks requires addressing several technical hurdles. The authors demonstrate that these distributions can be seamlessly incorporated into popular actor-critic architectures such as Soft Actor-Critic (SAC), GreedyAC, and Implicit Q-Learning (IQL).

Key contributions include the derivation of tractable log-likelihood gradients, enabling training via standard backpropagation. Furthermore, to overcome the lack of native support for $q$-Gaussians in standard deep learning libraries, the researchers implemented a Generalized Box-Muller Method for efficient sampling. For sparse policies ($q < 1$), they introduced an $L_2$-distance-based "nearest neighbor" mechanism to handle actions that fall outside the distribution's support during offline training, preventing undefined log-likelihood values.

Empirical Results and the "Drop-in" Winner

The proposed distributions were rigorously tested across classic control tasks and the D4RL MuJoCo offline RL benchmark. A standout finding was the exceptional performance of the Student’s t-distribution. It emerged as a superior "drop-in" replacement for the Gaussian, demonstrating greater stability across various hyperparameters and algorithms.

Overall performance across TAWAC and IQL algorithms on the D4RL benchmark. Heavy-tailed q-Gaussian and Student's t distributions significantly outperform the standard Squashed Gaussian baseline.

As shown in Figure 2, heavy-tailed variants—specifically the $q$-Gaussian ($q > 1$) and the Student’s t—consistently outperformed the standard "Squashed Gaussian" baseline. In offline settings, these distributions improved performance by up to 20% relative to the baseline. In online control, the Gaussian was the top performer in only 1 out of 12 tested settings, while the Student’s t or $q$-Gaussian dominated the rest.

Insights into Policy Evolution

Visualization of policy training revealed that standard Gaussians often collapse into a "delta-like" policy very early in the training process (within the first 10%). This premature convergence traps the agent in local optima. In contrast, $q$-exponential policies maintain stochasticity for longer periods, allowing them to escape local optima in environments with sparse rewards, such as the Mountain Car task.

Furthermore, the research highlighted the importance of matching the policy architecture to the mathematical objective. For instance, the Tsallis Advantage Weighted Actor-Critic (TAWAC) algorithm implicitly targets a $q$-Gaussian distribution. Using an explicit $q$-Gaussian policy with TAWAC led to significant performance gains by aligning the implementation with the underlying theory.

Limitations and Future Outlook

While the $q$-exponential family offers clear advantages, challenges remain. Sparse policies ($q < 1$) are particularly difficult to use in offline RL; if a behavior policy in a dataset contains an action that the sparse policy deems "impossible," the training objective becomes undefined. Additionally, sampling $q$-Gaussians is roughly 2-4 times slower than standard Gaussians, though this overhead is typically negligible compared to the total training time of deep RL models.

Future research will likely focus on better methods for projecting out-of-support actions onto the boundaries of sparse distributions and further exploring the learnable "degrees of freedom" in distributions like the Student’s t to automatically adapt tail heaviness during training.

Conclusion

The transition from the standard Gaussian to the $q$-exponential family represents a significant step toward more robust and explorative policy optimization. By embracing heavy-tailed distributions like the Student’s t, researchers can achieve more stable training and better performance across a wide array of reinforcement learning challenges.

Summary updated

BibTeX
@inproceedings{zhu2025q,
  author = {Lingwei Zhu and Haseeb Shah and Han Wang and Martha White},
  title = {{q-exponential family for policy optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=OyyE1FDdrQ},
}
Figure