TL;DR
This paper systematically explores the "critic" side of Soft Actor-Critic (SAC), testing whether learning advantages instead of Q-values, using multi-step targets, or training on replayed trajectory segments improves performance. The standout finding is that training a $\lambda$-return critic on replayed data (SAC-$\lambda$-R) significantly boosts performance and robustness in continuous control tasks, even without complex off-policy corrections.
What Problem Does This Paper Address?
Soft Actor-Critic (SAC) is a cornerstone of modern deep reinforcement learning, prized for its stability and exploration thanks to entropy regularization. However, most implementations stick to a "standard" critic: learning a soft Q-function using one-step temporal-difference (TD) errors from a replay buffer.
The authors argue that this design is more of a convention than a requirement. They set out to answer three fundamental questions:
- What to learn? Should we estimate the soft Q-value or the soft advantage?
- How deep to look? Does increasing the rollout depth (using multi-step $\lambda$-returns) help?
- Where to get data? Should multi-step targets be computed from fresh on-policy rollouts or replayed from the buffer?
Methods
The researchers evaluated six variants of SAC by varying three design dimensions, as illustrated in the training pipelines below:
- Axis 1: Target Type. They compared standard Q-learning against Soft Advantage (SAC-A) and Dueling-Advantage (SAC-DA) architectures.
- Axis 2: Rollout Depth. They moved beyond one-step TD to $\lambda$-returns (which interpolate between one-step and Monte Carlo returns) and Generalized Advantage Estimation (GAE).
- Axis 3: Data Source. They tested whether multi-step targets perform better when learned on-policy (fresh data) or from replayed segments (SAC-$\lambda$-R and SAC-GAE-R).
Crucially, they treated hyperparameters as part of the algorithm. Instead of reporting a single "tuned" result, they performed massive random hyperparameter searches (Sobol sampling) across 14 tasks to measure how robust each variant is to different settings.
Datasets
The study utilized 14 tasks across three popular benchmark suites:
- Classic Control: Acrobot, CartPole, MountainCar, and Pendulum (via
gymnax). - MinAtar: Simplified Atari-like environments including Freeway, Asterix, Breakout, and SpaceInvaders.
- Brax (MuJoCo-style): Continuous control tasks including Ant, HalfCheetah, Hopper, Humanoid, Reacher, and Walker2d.
These tasks cover both discrete and continuous action spaces, allowing the authors to see if their findings generalize across different problem types.
Key Insights and Ideas
- Advantages aren't a silver bullet for SAC: In algorithms like PPO, advantages reduce gradient variance. In SAC, the policy gradient is mathematically unaffected by state-dependent shifts in the critic. Empirically, learning soft advantages (SAC-A/DA) offered no benefit over Q-values.
- Replay beats "Freshness" for multi-step targets: Conventional wisdom suggests that multi-step targets from a replay buffer need "off-policy correction" (like importance sampling) because the data was generated by an old policy. This paper finds that the benefit of reusing data via replay far outweighs the bias introduced by the lack of correction.
- Simplicity wins with $\lambda$-returns: While GAE is a popular choice for advantage estimation, the authors found that the simpler soft $\lambda$-return critic performed just as well, making GAE's extra complexity unnecessary in the SAC framework.
- Continuous control loves depth + replay: The combination of multi-step targets and experience replay (SAC-$\lambda$-R) was the clear winner for continuous tasks, showing better peak performance and easier hyperparameter tuning.
Key Results
The primary results are captured in performance profiles, which show the probability of an algorithm exceeding a certain performance threshold across all tasks and hyperparameters.
- Continuous Control: SAC-$\lambda$-R was the top performer. It reached the highest peak scores on 5 out of 7 continuous tasks and was significantly more robust to hyperparameter choices than standard SAC.
- Discrete Tasks: No variant clearly outperformed standard SAC. The on-policy multi-step variants (SAC-$\lambda$ and SAC-GAE) actually performed significantly worse, likely due to lower sample efficiency compared to replay-based methods.
- Transferability: SAC-$\lambda$-R showed the best "transfer performance"—meaning a configuration tuned on one set of tasks was highly likely to work well on a new, unseen task.
Limitations and Future Work
- Limited Continuous Suite: While the results on Brax are strong, the authors acknowledge that 7 tasks is a small sample size given SAC's widespread use in continuous control.
- Confounded Replay Effects: The comparison between on-policy and replay-based variants bounds the combined effect of data reuse, larger batch sizes, and different update schedules. It doesn't perfectly isolate the effect of "target staleness" bias alone.
- Search Centering: The hyperparameter search was centered on a configuration already tuned for standard SAC, which might conservatively bias the results against the new variants.
- Integration: The authors studied these changes in isolation. A natural next step is to see how SAC-$\lambda$-R interacts with other common SAC extensions like distributional RL or automated temperature tuning.
Summary updated