Proximal Policy Optimization
- Also called: PPO, PPO-Clip
- Category: Learning & autonomy › RL & simulation
- Tags: reinforcement-learning, policy-gradient, on-policy, robot-learning, sim-to-real, locomotion
An on-policy reinforcement learning algorithm that improves a policy in small, clipped steps, widely used to train robot locomotion and control policies in parallel simulation.
Definition
Proximal Policy Optimization (PPO) is a policy gradient algorithm for reinforcement learning introduced by John Schulman and colleagues at OpenAI in 2017. It trains a stochastic policy, usually a neural network, by collecting batches of experience with the current policy and then updating the policy to make actions with high advantage more likely. Its central idea is to keep each update close to the policy that collected the data, which makes training more stable than plain policy gradient methods without the complexity of earlier trust-region methods such as TRPO.
The best-known form of PPO uses a clipped surrogate objective. For each action, it computes the ratio between the probability of that action under the new policy and under the old one. The objective multiplies this ratio by an advantage estimate but clips the ratio to a small interval around one, so the optimizer gains nothing from moving the policy too far in a single update. This lets PPO run several epochs of minibatch gradient descent on the same batch of data. Advantages are typically computed with generalized advantage estimation (GAE) using a learned value function, so PPO is an actor-critic method.
PPO is on-policy: data collected with an older policy is discarded after each update. This makes it less sample-efficient than off-policy methods such as Soft Actor-Critic, which reuse experience from a replay buffer. In robotics this weakness matters less when training in simulation, because GPU-based simulators can run thousands of environments in parallel and generate large batches of fresh data quickly. Its simplicity, stability and good scaling with parallel environments made PPO the default algorithm in many sim-to-real locomotion pipelines.
Results with PPO are sensitive to implementation details. Observation and advantage normalization, learning-rate schedules, entropy bonuses, value-function clipping and network initialization all affect performance. Robotics teams usually rely on well-tested implementations rather than writing their own, and they shape rewards and randomize simulation parameters carefully, since PPO will exploit any flaw in the reward or the simulator.
Why it matters
PPO is the algorithm behind a large share of the reinforcement learning results on real legged robots, humanoids and dexterous hands, because it tolerates a wide range of hyperparameters and scales well with massively parallel simulation. Understanding it is a practical prerequisite for reading or reproducing most sim-to-real robot learning work.
Key applications
- Legged locomotion — training walking and running policies for quadrupeds and humanoids in simulation
- Dexterous manipulation — learning in-hand object reorientation with multi-fingered hands
- Drone control — learning agile flight and racing policies
- Whole-body control — training policies that coordinate many joints at once
- Language model fine-tuning — PPO is also used in reinforcement learning from human feedback
In practice
A typical team defines the robot, its observations, actions and rewards in a GPU simulator such as Isaac Lab or MuJoCo, launches thousands of parallel environments, and trains a PPO policy with an existing library such as rsl_rl, Stable-Baselines3 or skrl. Each iteration collects a short rollout from every environment, computes advantages, and runs a few epochs of clipped updates. Engineers monitor episode reward, episode length, policy entropy and value loss, and most of their effort goes into reward terms, curriculum and domain randomization rather than into the algorithm itself. The trained policy, usually a small multilayer perceptron or recurrent network, is exported and run on the robot at a fixed control rate.
Common misconceptions
- PPO is the most sample-efficient RL algorithm — it is on-policy and discards old data; its strengths are stability and scaling with parallel simulation
- PPO works out of the box — performance depends heavily on reward design, normalization and other implementation details
- PPO is only for discrete actions — it handles continuous action spaces with Gaussian policies and is mostly used that way in robotics
- Clipping guarantees monotonic improvement — clipping is a heuristic that discourages large updates, not a formal trust-region guarantee
Who should know this
- Robot learning engineers training policies in simulation
- Locomotion researchers working on legged and humanoid robots
- Students learning deep reinforcement learning
- Simulation engineers building training environments
Related technologies
- Trust Region Policy Optimization (TRPO)
- Generalized Advantage Estimation (GAE)
- Soft Actor-Critic
- Isaac Lab
- MuJoCo
- Domain Randomization
Related terms
Related projects
- LeRobot: HuggingFace ML library for real-world robotics in PyTorch
- MuJoCo: DeepMind physics engine for robotics and ML research
- PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
- OpenPI: Physical Intelligence VLA foundation models for general robot control
- Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
- Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
Videos
- Proximal Policy Optimization Explained: Edan Meyer