$q$-exponential family for policy optimization

Lingwei Zhu, Haseeb Shah, Han Wang, Yukie Nagai, Martha White

In the world of Reinforcement Learning (RL), the Gaussian (Normal) distribution is the undisputed king of continuous action spaces. It’s mathematically elegant, easy to sample, and supported by every major deep learning library. But is it actually the best choice? A new paper suggests that we might be leaving significant performance on the table by sticking to the standard "bell curve."

TL;DR

This paper introduces the $q$-exponential family as a flexible, tractable alternative to Gaussian policies in reinforcement learning. By using heavy-tailed distributions like the Student’s t, the authors demonstrate improved stability and performance across a wide range of online and offline actor-critic algorithms.

What Problem Does This Paper Address?

Most modern RL algorithms for continuous actions (like walking robots or flying drones) represent their "policy" as a Gaussian distribution. While convenient, Gaussians have "thin tails," meaning they assign near-zero probability to actions far from the mean very quickly.

This leads to two major issues:

  1. Premature Convergence: Gaussian policies often "collapse" too quickly, losing the stochasticity needed for exploration before finding the optimal path.
  2. Sensitivity to Noise: Because they lack "heavy tails," Gaussians are less robust to outliers and noisy data, which are rampant in real-world RL.

The authors sought a broader family of distributions that could offer "heavy tails" (for better exploration and robustness) or "sparse tails" (for more decisive actions) while remaining computationally efficient.

Methods

The researchers turned to the $q$-exponential family, a concept borrowed from statistical physics. This family is governed by an entropic index $q$:

Comparison of Gaussian, light-tailed, and heavy-tailed distributions.
  • $q = 1$: Recovers the standard Gaussian distribution.
  • $q > 1$: Produces heavy-tailed distributions (like the Student’s t-distribution). These are "fat" at the edges, encouraging more diverse exploration.
  • $q < 1$: Produces light-tailed or sparse distributions. These have finite support, meaning they eventually hit zero probability, effectively "clipping" unlikely actions.

The authors derived the necessary gradients for these distributions to be used in deep RL and integrated them into several popular actor-critic algorithms, including Soft Actor-Critic (SAC), Implicit Q-Learning (IQL), and Tsallis Advantage Weighted Actor-Critic (TAWAC). They also provided a specialized sampling method (Generalized Box-Muller) to ensure these policies could be used efficiently in practice.

Datasets

The team evaluated their approach on two main fronts:

  1. Online Classic Control: Standard benchmarks like Mountain Car, Pendulum, and Acrobot.
  2. Offline RL (D4RL MuJoCo): The industry-standard suite for offline learning, featuring HalfCheetah, Hopper, and Walker2d tasks with varying data quality (Medium, Medium-Replay, and Medium-Expert).

Key Insights and Ideas

  • The Student’s t is a "Drop-in" Winner: The Student’s t-distribution (a specific $q$-exponential) proved to be remarkably stable. It features a learnable "degrees of freedom" parameter ($\nu$) that allows the policy to automatically decide how heavy its tails should be.
  • Matching the Objective: Some RL algorithms (like TAWAC) are mathematically derived using Tsallis entropy. The authors found that using a $q$-Gaussian policy explicitly matches the theoretical "target" of these algorithms, leading to much better performance than the standard Gaussian approximation.
  • Resisting Collapse: Visualizations of the learning process showed that Gaussian policies often collapse into a single point (a delta function) very early in training. In contrast, $q$-exponential policies maintained a healthy level of stochasticity for longer, preventing the agent from getting stuck in local optima.

Key Results

Overall performance of different policy distributions relative to the Squashed Gaussian baseline.
  • Superiority over Gaussian: In online tasks, the Student’s t-policy outperformed the Gaussian in 75% of the settings tested. In fact, the Gaussian was the "best" performer in only 1 out of 12 experimental configurations.
  • Offline Performance Boost: On the D4RL offline benchmarks, heavy-tailed policies (Student’s t and heavy $q$-Gaussian) consistently beat the "Squashed Gaussian" baseline, with relative improvements reaching up to ~20% on certain tasks and showing a clear aggregate advantage.
  • Stability: The Student’s t-distribution was found to be more robust to hyperparameter changes than the Gaussian, making it a safer "default" choice for researchers.

Limitations and Future Work

The primary challenge identified involves light-tailed ($q < 1$) policies in offline settings. Because these distributions have finite support, they can assign a probability of zero to certain actions. If an offline dataset contains an action that falls outside this "support," the math breaks (log-likelihood becomes $-\infty$).

While the authors proposed a "nearest-neighbor" fix to handle these out-of-support actions, it didn't fully solve the performance gap in offline RL. Future work will likely focus on more sophisticated projection methods to map dataset actions onto the boundaries of sparse $q$-Gaussian policies.

In conclusion, the paper makes a compelling case: if you are currently using a Gaussian policy for your RL agent, switching to a Student’s t-distribution might be the easiest performance upgrade you’ll find all year.

Summary updated

Abstract

Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family that remains tractable: the $q$-exponential family. This family of policies is flexible, allowing the specification of both heavy-tailed policies ($q>1$) and light-tailed policies ($q<1$). This paper examines the interplay between $q$-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed $q$-Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems. In summary, we find that the Student's t policy a strong candidate for drop-in replacement to the Gaussian. Our code is available at \url{https://github.com/lingweizhu/qexp}.
BibTeX
@inproceedings{zhu2025q,
  author = {Lingwei Zhu and Haseeb Shah and Han Wang and Yukie Nagai and Martha White},
  title = {{$q$-exponential family for policy optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=OyyE1FDdrQ},
}
Figure