Revisiting Critic Estimation in Soft Actor-Critic Algorithms

Jan Felix Kleuker, Koen Ponse, Aske Plaat, Thomas M. MoerlandarXiv 2026 · arXiv

TL;DR

This paper systematically explores the "critic" side of Soft Actor-Critic (SAC), testing whether learning advantages instead of Q-values, using multi-step targets, or training on replayed trajectory segments improves performance. The standout finding is that training a $\lambda$-return critic on replayed data (SAC-$\lambda$-R) significantly boosts performance and robustness in continuous control tasks, even without complex off-policy corrections.

What Problem Does This Paper Address?

Soft Actor-Critic (SAC) is a cornerstone of modern deep reinforcement learning, prized for its stability and exploration thanks to entropy regularization. However, most implementations stick to a "standard" critic: learning a soft Q-function using one-step temporal-difference (TD) errors from a replay buffer.

The authors argue that this design is more of a convention than a requirement. They set out to answer three fundamental questions:

  1. What to learn? Should we estimate the soft Q-value or the soft advantage?
  2. How deep to look? Does increasing the rollout depth (using multi-step $\lambda$-returns) help?
  3. Where to get data? Should multi-step targets be computed from fresh on-policy rollouts or replayed from the buffer?

Methods

The researchers evaluated six variants of SAC by varying three design dimensions, as illustrated in the training pipelines below:

Figure 2: Training pipelines of the SAC variants. (a) SAC-A and SAC-DA keep SAC’s replay-based one-step pipeline but learn soft advantages. (b) SAC trains its Q-network on replayed transitions, SAC-λ on parallel on-policy rollouts, and SAC-λ-R on replayed trajectory segments. (c) SAC-GAE and SAC-GAE-R learn a soft value network, on rollouts or replayed segments, and compute soft GAEs for the policy update.
  • Axis 1: Target Type. They compared standard Q-learning against Soft Advantage (SAC-A) and Dueling-Advantage (SAC-DA) architectures.
  • Axis 2: Rollout Depth. They moved beyond one-step TD to $\lambda$-returns (which interpolate between one-step and Monte Carlo returns) and Generalized Advantage Estimation (GAE).
  • Axis 3: Data Source. They tested whether multi-step targets perform better when learned on-policy (fresh data) or from replayed segments (SAC-$\lambda$-R and SAC-GAE-R).

Crucially, they treated hyperparameters as part of the algorithm. Instead of reporting a single "tuned" result, they performed massive random hyperparameter searches (Sobol sampling) across 14 tasks to measure how robust each variant is to different settings.

Datasets

The study utilized 14 tasks across three popular benchmark suites:

  • Classic Control: Acrobot, CartPole, MountainCar, and Pendulum (via gymnax).
  • MinAtar: Simplified Atari-like environments including Freeway, Asterix, Breakout, and SpaceInvaders.
  • Brax (MuJoCo-style): Continuous control tasks including Ant, HalfCheetah, Hopper, Humanoid, Reacher, and Walker2d.

These tasks cover both discrete and continuous action spaces, allowing the authors to see if their findings generalize across different problem types.

Key Insights and Ideas

  • Advantages aren't a silver bullet for SAC: In algorithms like PPO, advantages reduce gradient variance. In SAC, the policy gradient is mathematically unaffected by state-dependent shifts in the critic. Empirically, learning soft advantages (SAC-A/DA) offered no benefit over Q-values.
  • Replay beats "Freshness" for multi-step targets: Conventional wisdom suggests that multi-step targets from a replay buffer need "off-policy correction" (like importance sampling) because the data was generated by an old policy. This paper finds that the benefit of reusing data via replay far outweighs the bias introduced by the lack of correction.
  • Simplicity wins with $\lambda$-returns: While GAE is a popular choice for advantage estimation, the authors found that the simpler soft $\lambda$-return critic performed just as well, making GAE's extra complexity unnecessary in the SAC framework.
  • Continuous control loves depth + replay: The combination of multi-step targets and experience replay (SAC-$\lambda$-R) was the clear winner for continuous tasks, showing better peak performance and easier hyperparameter tuning.

Key Results

The primary results are captured in performance profiles, which show the probability of an algorithm exceeding a certain performance threshold across all tasks and hyperparameters.

Performance profiles showing that SAC-λ-R (light blue) dominates in continuous tasks compared to standard SAC and other variants.
  • Continuous Control: SAC-$\lambda$-R was the top performer. It reached the highest peak scores on 5 out of 7 continuous tasks and was significantly more robust to hyperparameter choices than standard SAC.
  • Discrete Tasks: No variant clearly outperformed standard SAC. The on-policy multi-step variants (SAC-$\lambda$ and SAC-GAE) actually performed significantly worse, likely due to lower sample efficiency compared to replay-based methods.
  • Transferability: SAC-$\lambda$-R showed the best "transfer performance"—meaning a configuration tuned on one set of tasks was highly likely to work well on a new, unseen task.

Limitations and Future Work

  • Limited Continuous Suite: While the results on Brax are strong, the authors acknowledge that 7 tasks is a small sample size given SAC's widespread use in continuous control.
  • Confounded Replay Effects: The comparison between on-policy and replay-based variants bounds the combined effect of data reuse, larger batch sizes, and different update schedules. It doesn't perfectly isolate the effect of "target staleness" bias alone.
  • Search Centering: The hyperparameter search was centered on a configuration already tuned for standard SAC, which might conservatively bias the results against the new variants.
  • Integration: The authors studied these changes in isolation. A natural next step is to see how SAC-$\lambda$-R interacts with other common SAC extensions like distributional RL or automated temperature tuning.

Summary updated

Abstract

Soft Actor-Critic (SAC) is a widely used deep reinforcement learning algorithm that combines entropy regularization with an actor-critic architecture. Standard implementations rely on one-step temporal-difference learning of Q-functions, yet alternative critic design choices remain underexplored. In this work, we revisit three such choices: learning soft advantages instead of soft Q-values, the use of multi-step return targets, and the training of multi-step critics from replay buffer data. Keeping the policy objective fixed, we compare the six critic variants these choices generate: (i) advantage and (ii) dueling-advantage critics learned from one-step transitions, and $λ$-return and soft Generalized Advantage Estimation (GAE) critics learned either (iii, iv) on-policy or (v, vi) from replayed trajectory segments. Each variant is evaluated across an entire random hyperparameter search on fourteen tasks, treating the hyperparameters as part of the algorithm. Learning soft advantages instead of soft Q-values does not improve performance, and the soft GAE critic adds nothing over the simpler $λ$-return critic. What does improve performance, most clearly on continuous control, is training a $λ$-return critic on replayed trajectory segments, even without importance weighting or trace cutting. This extends earlier evidence that uncorrected multi-step targets work well under replay to the full soft $λ$-return, entropy terms included. By design, the underlying ideas studied here extend beyond SAC, and our empirical findings aim to encourage their application to a broader class of reinforcement learning algorithms.
BibTeX
@misc{kleuker2026revisiting,
  author = {Jan Felix Kleuker and Koen Ponse and Aske Plaat and Thomas M. Moerland},
  title = {{Revisiting Critic Estimation in Soft Actor-Critic Algorithms}},
  year = {2026},
  eprint = {2611.05666},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05666},
}
Figure