Asymmetric Actor-Critic

Also called: asymmetric critic, privileged critic
Category: Learning & autonomy › RL & simulation
Tags: reinforcement-learning, sim-to-real, privileged-information, actor-critic, robot-learning

An actor-critic reinforcement learning setup in which the critic is trained on privileged simulator state while the actor uses only observations the real robot will have.

Definition

Asymmetric actor-critic is a way of training a policy with an actor-critic reinforcement learning algorithm when the training environment knows more than the deployed robot will. The actor, which becomes the deployed policy, receives only realistic observations such as camera images, joint encoders or noisy object pose estimates. The critic, which estimates the value of states and exists only to reduce the variance of the policy gradient during training, is given extra privileged information: exact object poses and velocities, contact forces, friction coefficients and other simulator state. Pinto and colleagues introduced the term in 2018 for image-based robot learning, where the actor saw images and the critic saw full state.

The idea works because the critic is thrown away after training. Nothing it consumes has to be measured on the real robot, so it can be given whatever makes value estimation easiest. A critic with full state learns faster and more accurately than one that must infer hidden quantities from partial observations, which gives the actor better learning signals. The approach fits naturally with on-policy methods such as PPO, used for in-hand reorientation in Isaac Gym, and with off-policy methods such as DDPG or SAC.

Asymmetric actor-critic differs from teacher-student training. In teacher-student training a separate teacher policy is first trained with privileged inputs and then distilled into a student that sees only real observations. In asymmetric actor-critic there is one policy, trained directly by reinforcement learning on realistic inputs; only the value function is privileged. The two are often combined. One subtlety is that under partial observability a critic that conditions only on the true state can give biased value estimates for a policy that acts on observation history, so many implementations feed the critic both the privileged state and the actor's observations.

Why it matters

Sim-to-real reinforcement learning needs policies that run on real sensors, but learning from partial, noisy observations is slow and unstable. Giving the critic privileged simulator state is a cheap change to standard algorithms that often makes training much faster and more reliable, without adding anything the deployed robot must measure.

Key applications

In-hand object reorientation with dexterous robot hands
Image-based manipulation policies trained in simulation
Legged locomotion policies trained with privileged terrain and contact information
Sim-to-real reinforcement learning with GPU-parallel simulators
Partially observable control tasks where the simulator exposes hidden state

In practice

A team trains a multi-fingered hand to reorient objects in a GPU simulator with PPO. The actor network receives fingertip positions, joint angles and an object pose estimate with added noise and latency, matching what a camera-based pose tracker will deliver on hardware. The critic network receives the same inputs plus the exact object pose, linear and angular velocity, the goal orientation and the randomized friction and mass values for that environment. Both networks share the same rollouts and are updated together. At deployment only the actor's weights are exported to the robot computer; the critic and the simulator-only inputs are never needed again.

Common misconceptions

The deployed policy uses the privileged state — only the critic sees it during training, and the critic is discarded at deployment
It is the same as teacher-student training — teacher-student trains a privileged policy and distills it, while asymmetric actor-critic trains one realistic policy directly with a privileged value function
A state-only critic is always correct — under partial observability it can bias the value estimate for an observation-based actor, which is why critics are often given the actor's observations as well

Who should know this

Reinforcement learning engineers working on sim-to-real transfer
Dexterous manipulation researchers
Legged locomotion developers training policies in simulation
Simulation engineers designing observation spaces for training

Related technologies

Proximal Policy Optimization
Isaac Gym
Privileged Learning
Domain Randomization
Soft Actor-Critic

Related terms

Related projects

LeRobot: HuggingFace ML library for real-world robotics in PyTorch
MuJoCo: DeepMind physics engine for robotics and ML research
PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
OpenPI: Physical Intelligence VLA foundation models for general robot control
Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim