Soft Actor-Critic

• Also called: SAC
• Category: Learning & autonomy › RL & simulation
• Tags: reinforcement-learning, off-policy, actor-critic, maximum-entropy, robot-learning, continuous-control

An off-policy reinforcement learning algorithm that trains a stochastic policy to maximize both reward and action entropy, reusing past experience from a replay buffer for sample efficiency.

Definition

Soft Actor-Critic (SAC) is an off-policy, actor-critic reinforcement learning algorithm for continuous action spaces, introduced by Tuomas Haarnoja and colleagues in 2018. It is built on the maximum entropy framework: instead of maximizing expected reward alone, the policy maximizes reward plus a bonus for the entropy, or randomness, of its actions. A temperature parameter sets the balance between the two. The entropy term encourages broad exploration and discourages the policy from collapsing early onto a single behavior.

SAC learns a stochastic policy (the actor), usually a Gaussian whose samples are squashed with a tanh function to respect action limits, and two Q-function critics. Taking the minimum of the two critics when computing learning targets reduces the overestimation bias common in Q-learning, a technique shared with the TD3 algorithm, and slowly updated target networks stabilize learning. A later version of SAC adjusts the temperature automatically to keep the policy's entropy near a chosen target, removing one of the most sensitive hyperparameters.

Because SAC is off-policy, every transition the robot experiences is stored in a replay buffer and reused for many gradient updates. This makes it considerably more sample-efficient than on-policy methods such as PPO, which discard data after each update. That property made SAC a common choice for learning directly on physical robots, where every minute of interaction costs time and wear, and for tasks where simulation is slow or unavailable.

The trade-offs lie in compute and scaling. SAC performs many network updates per environment step, and it has historically benefited less than PPO from massively parallel GPU simulation, where fresh data is cheap. Training can also become unstable in high-dimensional action spaces or with large networks, and ongoing research studies how to scale off-policy methods such as SAC to many parallel environments and to robots with many joints.

Why it matters

SAC made deep reinforcement learning practical in settings where data is expensive, including learning directly on real robot hardware. It remains one of the standard baselines for continuous control, and its entropy-regularized formulation influences many later off-policy and model-based methods.

Key applications

• Learning on real hardware — training manipulation and locomotion skills with limited robot time
• Continuous control benchmarks — a standard baseline for simulated robot tasks
• Manipulation — learning grasping, pushing and insertion policies with sparse or shaped rewards
• Fine-tuning pretrained policies — improving behavior-cloned policies with online reinforcement learning
• Vehicle and drone control — learning continuous steering and thrust commands

In practice

A team using SAC typically wraps the robot or simulator in a standard environment interface, picks a library implementation such as Stable-Baselines3, and sets the replay buffer size, batch size, learning rates and the ratio of gradient updates to environment steps. When learning on hardware, they add safety limits on actions, automatic resets and reward signals that can be measured from onboard sensors, and they often seed the replay buffer with demonstrations or data from earlier runs. Engineers watch the critic loss, the temperature value and the policy's entropy; a temperature that falls to near zero or critic estimates that grow without bound are common signs that training needs adjustment.

Common misconceptions

• SAC is always better than PPO — SAC is more sample-efficient, but PPO often trains faster in wall-clock time when thousands of simulated environments run in parallel
• The entropy bonus makes the final policy random — at deployment the mean action is usually used, and automatic temperature tuning lowers entropy as learning progresses
• SAC handles discrete actions unchanged — the standard algorithm targets continuous actions; discrete variants require modifications
• Off-policy means any old data works — data far from the current policy's behavior can still cause unstable value estimates, especially in fully offline settings

Who should know this

• Robot learning researchers training on physical hardware
• Reinforcement learning practitioners working on continuous control
• Manipulation engineers using online fine-tuning
• Students learning actor-critic methods

Related technologies

• Proximal Policy Optimization
• TD3
• Deep Deterministic Policy Gradient (DDPG)
• Replay Buffer
• Maximum Entropy Reinforcement Learning
• Stable-Baselines3

Related terms

• Reinforcement Learning
• Proximal Policy Optimization
• Sim2Real

Related projects

• LeRobot: HuggingFace ML library for real-world robotics in PyTorch
• MuJoCo: DeepMind physics engine for robotics and ML research
• PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
• OpenPI: Physical Intelligence VLA foundation models for general robot control
• Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
• Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim

Videos

• L5 DDPG and SAC (Foundations of Deep RL Series): Pieter Abbeel