Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
核心贡献 · Key contributions
提出软演员-评论家（SAC），一种基于最大熵的离策略深度强化学习算法。 Proposes soft actor-critic (SAC), an off-policy maximum entropy deep RL algorithm.
将离策略更新与稳定的随机演员-评论家公式相结合。 Combines off-policy updates with a stable stochastic actor-critic formulation.
在连续控制基准上取得最先进性能，优于先前的在策略和离策略方法。 Achieves state-of-the-art performance on continuous control benchmarks, outperforming prior on-policy and off-policy methods.
与其他离策略算法不同，展示了跨不同随机种子的高度稳定性。 Demonstrates high stability across different random seeds, unlike other off-policy algorithms.
在最大熵框架下提供了软策略迭代的收敛性证明。 Provides convergence proof for soft policy iteration in the maximum entropy framework.
使用两个 Q 函数来减轻正向偏差并提高训练速度。 Uses two Q-functions to mitigate positive bias and improve training speed.
局限 · Limitations
对奖励缩放敏感，该缩放控制温度和策略随机性。 Sensitive to reward scaling, which controls the temperature and policy stochasticity.
需要调整奖励缩放作为关键超参数以获得最佳性能。 Requires tuning of reward scale as a key hyperparameter for optimal performance.
假设连续动作空间；未直接解决离散动作的适用性。 Assumes continuous action spaces; applicability to discrete actions not directly addressed.
由于探索-利用平衡，奖励幅度非常小或非常大时性能可能下降。 Performance may degrade with very small or very large reward magnitudes due to exploration-exploitation balance.
理论收敛性证明假设有限动作空间，限制了直接扩展到连续动作。 Theoretical convergence proof assumes finite action space, limiting direct extension to continuous actions.
论文章节 · Sections（共 14）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 预备知识3 Preliminaries
3.1 符号说明3.1 Notation
3.2 最大熵强化学习3.2 Maximum Entropy Reinforcement Learning
4 从软策略迭代到软演员-评论家4 From Soft Policy Iteration to Soft Actor-Critic
4.1 软策略迭代的推导4.1 Derivation of Soft Policy Iteration