Actor-Critic

Also called: actor-critic methods, actor-critic algorithm
Category: Learning & autonomy › RL & simulation
Tags: reinforcement-learning, policy-gradient, value-function, deep-learning, robot-learning

A family of reinforcement learning methods that train two parts together: an actor, the policy that picks actions, and a critic, a learned value function used to evaluate and improve those actions.

Definition

Actor-critic methods are a family of reinforcement learning algorithms that combine policy learning with value learning. The actor is the policy, a function that maps the current state or observation to an action or to a distribution over actions. The critic is a learned estimate of how good states or actions are: a state-value function V(s), an action-value function Q(s, a), or an advantage function that measures how much better an action is than the policy's average. The critic is trained from experience, usually with temporal-difference learning, and the actor is updated in the direction the critic indicates will raise expected return.

The idea dates back to Barto, Sutton and Anderson's 1983 work on learning to balance a pole, and Konda and Tsitsiklis gave a formal analysis of actor-critic algorithms and their convergence around 2000. The motivation is variance reduction. Pure policy-gradient methods such as REINFORCE weight each action by the full observed return, which is very noisy; replacing that return with a critic's estimate, or subtracting a learned baseline to form an advantage, gives far lower-variance updates at the cost of some bias from the critic's errors. Techniques such as generalized advantage estimation let the user tune that bias-variance trade-off.

Most deep reinforcement learning used in robotics is actor-critic. On-policy variants such as A2C and proximal policy optimization (PPO) train a stochastic actor with a learned value baseline and are widely used for locomotion in massively parallel simulation. Off-policy variants such as DDPG, TD3 and soft actor-critic (SAC) learn a Q-function from a replay buffer and update the actor by following the critic's gradient, which uses data more efficiently and suits training on real hardware. Because the critic is only needed during training, it can use information the deployed actor never sees, which is the basis of asymmetric actor-critic training.

Why it matters

Actor-critic structure is what makes deep reinforcement learning stable enough for robot control. Knowing which part does what explains most practical tuning decisions, from value-loss weights and advantage normalization to why a critic can be given privileged simulator state, and it is the shared vocabulary behind PPO, SAC and their variants.

Key applications

Legged locomotion policies trained in simulation with PPO
Sample-efficient learning on real robot arms with SAC or TD3
Dexterous in-hand manipulation with privileged-information critics
Drone and quadrotor control policies
Camera and viewpoint control learned from reward signals
Fine-tuning policies pretrained by imitation learning

In practice

A team trains a quadruped walking policy with PPO in a GPU simulator running thousands of environments in parallel. The actor is a small multilayer perceptron that takes proprioceptive observations and outputs the mean of a Gaussian over joint position targets. The critic is a separate network that receives the same observations plus privileged terms such as true base velocity, terrain heights and friction, and predicts the expected discounted return. After each rollout, generalized advantage estimation combines the critic's predictions with observed rewards, the actor is updated with the clipped PPO objective, and the critic is regressed toward the computed returns. Engineers watch the critic's explained variance: if it stays low, advantages are mostly noise and the actor stops improving. Only the actor is exported to the robot's onboard computer.

Common misconceptions

The critic is part of the deployed controller — only the actor runs on the robot; the critic exists to train it
The critic's values are accurate measures of task success — they are learned approximations that can be biased, and Q-function overestimation is a known failure that TD3 and SAC counter with twin critics
PPO is not an actor-critic method — standard PPO implementations train a value function as a baseline for advantages, which makes them actor-critic
Actor-critic names one algorithm — it is a family that includes on-policy and off-policy, stochastic and deterministic methods

Who should know this

Reinforcement learning practitioners working on physical robots
Locomotion and manipulation engineers using learned controllers
Robot learning researchers designing training pipelines
Students moving from classical control to learned control

Related technologies

A2C and A3C
DDPG
TD3
Generalized Advantage Estimation
Stable-Baselines3
Isaac Lab

Related terms

Related projects

LeRobot: HuggingFace ML library for real-world robotics in PyTorch
MuJoCo: DeepMind physics engine for robotics and ML research
PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
OpenPI: Physical Intelligence VLA foundation models for general robot control
Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim

Videos

Actor Critic Algorithms: Siraj Raval
Everything You Need To Master Actor Critic Methods | Tensorflow 2 Tutorial: Machine Learning with Phil