Inverse Reinforcement Learning

• Also called: IRL, inverse RL
• Category: Learning & autonomy › Learning from demonstration
• Tags: reward-learning, imitation-learning, reinforcement-learning, robot-learning, learning-from-demonstration

Learning the reward function that explains an expert's demonstrated behavior, so that a policy can then be trained to optimize that reward with reinforcement learning or planning.

Definition

Inverse reinforcement learning (IRL) reverses the usual reinforcement learning problem. Instead of being given a reward and asked for a good policy, the learner is given demonstrations of good behavior and asked for a reward function under which that behavior is optimal or close to it. The learned reward is then used to train a policy, sometimes in new situations or on a different robot, on the assumption that the reward captures the intent of the task better than the demonstrated actions do. Andrew Ng and Stuart Russell gave an early formal treatment of the problem in 2000.

The problem is ill-posed: many reward functions explain the same behavior, including a reward that is zero everywhere. Methods resolve this ambiguity in different ways. Apprenticeship learning matches the expected feature counts of the learner to those of the expert. Maximum entropy IRL, introduced by Brian Ziebart and colleagues in 2008, models the expert as choosing trajectories with probability proportional to the exponential of their reward, which gives a well-defined likelihood and tolerates imperfect demonstrations. Adversarial methods train a discriminator to tell expert data from the policy's own data and use its output as a training signal; GAIL does this without returning an explicit reward, while AIRL is designed to recover one.

Classic IRL alternates between updating the reward and solving a full reinforcement learning problem under it, which is expensive. Practical robot systems therefore often use simpler reward-learning schemes, such as rewards based on similarity to demonstration frames or classifiers trained on examples of success, while keeping the core idea of deriving the training signal from examples. In control theory, the closely related problem of recovering a cost function from observed optimal trajectories is called inverse optimal control.

Why it matters

Writing rewards by hand for robot tasks is slow and error-prone, and policies exploit any gap between the reward and the intended behavior. IRL obtains rewards from demonstrations instead, which can carry over to conditions where copying the demonstrated actions would fail, and it links imitation learning with reinforcement learning.

Key applications

• Learning rewards for manipulation tasks from demonstrations
• Modeling driver and pedestrian behavior for autonomous vehicles
• Learning navigation cost maps from expert paths
• Inferring human preferences in human-robot interaction
• Providing training rewards for reinforcement learning on real robots

In practice

A team wants a robot arm to wipe a table the way a person does. They record a set of teleoperated demonstrations, then train a reward model alongside the policy: a discriminator learns to score state transitions as expert-like or not, and the policy is trained with reinforcement learning in simulation to maximize that score before being fine-tuned on the robot. To stop the policy from exploiting flaws in the learned reward, they regularize the discriminator, add a small hand-written term for task completion and review rollouts by eye. Later, when they move the task to a table of a different height, the learned reward is used to score the adapted policy's rollouts.

Common misconceptions

• IRL is the same as behavior cloning — behavior cloning copies actions directly, while IRL recovers a reward and then trains a policy to optimize it
• The recovered reward is the true reward — many rewards explain the same demonstrations, and IRL returns one that fits the data under its own assumptions
• IRL needs perfect demonstrations — early formulations assumed optimal experts, but maximum entropy and later methods tolerate noisy or suboptimal demonstrations

Who should know this

• Robot learning researchers
• Reinforcement learning engineers designing reward functions
• Autonomous driving behavior modeling teams
• Human-robot interaction researchers

Related technologies

• Maximum Entropy IRL
• Generative Adversarial Imitation Learning (GAIL)
• Adversarial Inverse Reinforcement Learning (AIRL)
• Inverse Optimal Control
• Apprenticeship Learning

Related terms

• Reinforcement Learning
• Imitation Learning
• Behavior Cloning
• DAgger
• Residual Reinforcement Learning

Related projects

• LeRobot: HuggingFace ML library for real-world robotics in PyTorch
• MuJoCo: DeepMind physics engine for robotics and ML research
• PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
• OpenPI: Physical Intelligence VLA foundation models for general robot control
• Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
• Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim

Videos

• CS 285: Lecture 20, Inverse Reinforcement Learning, Part 1: RAIL
• Inverse Reinforcement Learning Explained: Edan Meyer