Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

Reward Shaping

  • Category: Learning & autonomy › RL & simulation
  • Tags: reinforcement-learning, reward-design, sparse-rewards, robot-learning, simulation

Adding extra terms to a reinforcement-learning reward so the agent gets useful feedback before it first reaches the goal, ideally without changing which behavior is optimal.

Definition

Reward shaping modifies the reward function of a reinforcement-learning problem to make learning easier. Many robot tasks have a natural reward that is sparse: the agent receives a signal only when the drawer is open or the peg is inserted. With random exploration, a robot may almost never reach that state and so receives almost no learning signal. Shaping adds intermediate terms, such as the negative distance from the gripper to the handle, a bonus for a stable grasp, or progress through a sequence of subgoals, so that the agent receives feedback long before it first succeeds.

The risk is that the shaped reward changes the task. A robot rewarded for gripper proximity may learn to hover near the handle without pulling it, and a reward for forward velocity can produce gaits that exploit flaws in the simulator. Ng, Harada and Russell (1999) showed that shaping terms of a specific form, F(s, s′) = γΦ(s′) − Φ(s) for a potential function Φ defined over states, leave the optimal policies of the original problem unchanged. This is called potential-based reward shaping. Because any reward collected by moving through a cycle of states sums to zero under such terms, the agent cannot profit from loops that do not advance the task. Terms not of this form can change the optimal policy. That is sometimes intended, when the extra terms express real preferences such as smooth motion or low energy use, but it should be a deliberate choice.

In robotics practice, rewards for locomotion and manipulation often combine many weighted terms, and tuning their weights takes a large share of the engineering effort. Complements and alternatives include curriculum learning, hindsight relabeling of goals, learning from demonstrations, inverse reinforcement learning, and rewards computed from learned visual representations, for example an estimate of progress toward a subgoal image. These reduce manual reward design but bring failure modes of their own.

Why it matters

Whether a robot learning task converges at all often depends on its reward. Shaping is how most simulated locomotion and manipulation policies are made trainable, and badly shaped rewards are a common reason for policies that score well during training yet do not perform the intended task.

Key applications

  • Training manipulation policies when task success is sparse
  • Legged locomotion rewards combining velocity tracking, posture and energy terms
  • Long-horizon tasks split into subgoals with progress rewards
  • Navigation with distance-to-goal terms
  • Dexterous in-hand manipulation trained in simulation

In practice

A team trains a policy in simulation to open a cabinet drawer. With a reward given only when the drawer is open, training makes no progress. They add a term for the negative distance from the gripper to the handle, a bonus each time the handle is grasped, and a term proportional to how far the drawer has opened. The policy soon learns to reach and grasp but then stalls, because releasing and regrasping the handle collects the grasp bonus again and again. They replace the bonus with a potential-based term, the step-to-step change in a potential that rises as the drawer opens and as the gripper nears the handle, so that cycling between states earns nothing overall, and add a small penalty on joint velocity for smoother motion. They judge the final policy by the sparse success criterion, not by the shaped return.

Common misconceptions

  • Any added reward term is harmless if it points toward the goal — terms that are not potential-based can change the optimal policy and invite reward hacking
  • A higher shaped return means a better policy — only performance on the true task objective shows whether the policy does what was intended
  • Potential-based shaping always speeds up learning — it preserves the optimal policies, but a poorly chosen potential can still slow exploration
  • Reward shaping and curriculum learning are the same — shaping changes the reward, while a curriculum changes the order or difficulty of the tasks and environments

Who should know this

  • Reinforcement-learning engineers training robot policies
  • Locomotion and manipulation researchers working in simulation
  • Simulation engineers setting up training environments
  • Students learning reinforcement learning for robotics

Related technologies

  • Potential-Based Reward Shaping
  • Sparse Rewards
  • Hindsight Experience Replay
  • Curriculum Learning
  • Inverse Reinforcement Learning
  • Learned Reward Models

Related terms

  • Reinforcement Learning
  • Markov Decision Process (MDP)
  • Curriculum Learning
  • Hindsight Experience Replay (HER)
  • Inverse Reinforcement Learning

Related projects

  • GENESIS: World's fastest physics engine: 43M FPS for embodied AI learning
  • LeRobot: HuggingFace ML library for real-world robotics in PyTorch
  • MuJoCo: DeepMind physics engine for robotics and ML research
  • PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
  • OpenPI: Physical Intelligence VLA foundation models for general robot control
  • Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware

Videos

  • Reward Shaping: Udacity

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.