Residual Reinforcement Learning

Also called: residual RL, residual policy learning
Category: Learning & autonomy › RL & simulation
Tags: reinforcement-learning, robot-learning, manipulation, contact-rich-tasks, sim-to-real

Reinforcement learning of a correction that is added to the actions of an existing controller or learned policy, so that learning starts from reasonable behavior instead of from scratch.

Definition

Residual reinforcement learning splits a robot's action into two parts: the output of a base policy and a learned residual. The base policy can be a hand-designed controller, a trajectory follower or a policy trained by imitation learning. The residual policy observes the state, often along with the base action, and outputs a correction that is added to the base action. Only the residual is trained with reinforcement learning; the base policy stays fixed. The approach was introduced independently by two groups in 2018 and 2019, under the names residual policy learning and residual reinforcement learning for robot control.

Starting from a working base policy changes the learning problem in useful ways. Early in training, when the residual is near zero, the robot already behaves sensibly, so exploration is safer and rewards are easier to reach, which matters for sparse-reward tasks and for training on real hardware. The residual only has to learn what the base policy gets wrong, such as contact effects, friction or object variations that a controller does not model. Initializing the residual network to output zero and bounding its magnitude are common ways to keep early behavior close to the base policy.

The base policy also limits what can be learned. If it is far from good behavior, a bounded residual may not be able to fix it, and an unbounded one gives up the safety benefit. Variants let the residual adjust controller parameters such as gains or setpoints instead of raw actions, or combine residuals with policies learned from a small number of demonstrations, so that imitation provides the base behavior and reinforcement learning refines it for precision or robustness.

Why it matters

Pure reinforcement learning on real robots is slow and can be unsafe, while hand-designed controllers and imitation policies are often nearly right but fail on the details of contact. Residual RL combines the two, making it a practical way to improve existing controllers with a limited amount of real-world interaction.

Key applications

Contact-rich assembly such as peg-in-hole insertion
Refining imitation-learned policies with reinforcement learning
Correcting model-based controllers for unmodeled friction and dynamics
Adapting simulation-trained controllers to real hardware
Improving grasping and in-hand manipulation from a scripted baseline

In practice

A team uses an impedance controller to insert a peg into a tight-fitting hole. The controller reaches the hole reliably but often jams because of small pose errors and friction. They add a residual policy that sees the end-effector pose and the wrist force-torque reading and outputs a small offset to the controller's target position, clipped to a narrow range. The residual starts at zero, so the first episodes behave like the original controller, and it is trained with an off-policy algorithm such as soft actor-critic using a reward based on insertion depth. Over training, it learns the small corrective motions that free a jammed peg, while the base controller still handles the gross motion.

Common misconceptions

Residual RL retrains the base controller — the base policy stays fixed and only the additive correction is learned
Any base policy will do — the residual is usually bounded, so a poor base policy limits what the combined system can achieve
Residual RL removes the need for reward design — the residual still needs a reward that captures the task; the base policy only makes exploration easier and safer

Who should know this

Robot learning researchers
Manipulation engineers working on contact-rich tasks
Control engineers improving existing controllers with data
Sim-to-real practitioners

Related technologies

Soft Actor-Critic
Twin Delayed DDPG (TD3)
Impedance Control
Behavior Cloning
Proximal Policy Optimization

Related terms

Related projects

LeRobot: HuggingFace ML library for real-world robotics in PyTorch
MuJoCo: DeepMind physics engine for robotics and ML research
PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
OpenPI: Physical Intelligence VLA foundation models for general robot control
Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim