Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

Hindsight Experience Replay (HER)

  • Also called: HER
  • Category: Learning & autonomy › RL & simulation
  • Tags: reinforcement-learning, sparse-rewards, goal-conditioned, off-policy, manipulation

A technique for goal-conditioned reinforcement learning that relabels failed episodes with goals the agent actually reached, turning sparse-reward failures into useful training data.

Definition

Hindsight experience replay (HER) addresses a common problem in robot learning: with a sparse reward that is only given when a goal is reached, a randomly acting robot almost never succeeds, so it receives no learning signal. HER, introduced by Andrychowicz et al. in 2017, reuses each failed episode by pretending, after the fact, that the goal had been whatever the agent actually achieved. An episode that tried to push a block to point A but left it at point B becomes a successful example of pushing the block to point B.

The method works with goal-conditioned policies and value functions, which take both the state and the desired goal as input, and with off-policy algorithms such as DDPG, TD3 or SAC that learn from a replay buffer. Each stored transition is copied with a substituted goal and a reward recomputed for that goal. Several strategies choose the substitute: the final state of the episode, a random state from the same episode, or, most commonly, a state reached later in the same episode (the "future" strategy). Because the reward must be recomputed for any goal, HER requires a reward function that can be evaluated for arbitrary goals, typically whether the achieved goal lies within a tolerance of the desired one.

HER acts as an implicit curriculum: early on, the relabeled goals are easy ones near where the robot already ends up, and as the policy improves, the goals it reaches spread out to cover the space. It does not help when the goal cannot be read off the achieved state, when rewards depend on more than the final outcome, or with on-policy algorithms that cannot reuse old data. Relabeling also biases the goal distribution in the buffer, which later work has examined and corrected in various ways.

Why it matters

Sparse success rewards are the most natural way to specify many manipulation tasks, but they are very hard to learn from directly. HER made it practical to train pushing, sliding and pick-and-place policies from binary rewards without hand-shaped reward terms, and goal relabeling has since become a standard tool in goal-conditioned learning.

Key applications

  • Sparse-reward pushing, sliding and pick-and-place with robot arms
  • Training goal-conditioned policies that reach arbitrary target positions
  • Learning navigation to goal locations from success-only rewards
  • Relabeling demonstrations or play data with the goals they actually reached
  • Multi-goal benchmarks for off-policy reinforcement learning

In practice

An engineer trains a simulated arm to place a block at randomly chosen target positions, with a reward of 0 on success and -1 otherwise. Without HER, SAC barely improves, because the block rarely lands on the target by chance. With HER, each transition is also stored with goals drawn from block positions reached later in that episode, and the reward is recomputed as success if the block is within a few centimeters of the substituted goal. The engineer checks that the observation clearly separates the achieved goal (the block position) from the desired goal, so the relabeling code can swap one for the other.

Common misconceptions

  • HER works with any reinforcement learning algorithm — it relies on reusing stored transitions, so it fits off-policy methods such as DDPG, TD3 and SAC, not on-policy methods such as PPO
  • HER removes the need for any reward design — it still needs a reward that can be computed for any goal and a clear definition of which part of the state counts as the achieved goal
  • Relabeled data is unbiased — substituting achieved goals changes the distribution of goals and outcomes in the buffer, which can bias value estimates

Who should know this

  • Reinforcement learning practitioners working on manipulation
  • Robot learning researchers using sparse or binary rewards
  • Engineers building goal-conditioned policies
  • Students implementing off-policy learning algorithms

Related technologies

  • Deep Deterministic Policy Gradient (DDPG)
  • Twin Delayed DDPG (TD3)
  • Replay Buffer
  • Universal Value Function Approximators
  • Gymnasium-Robotics

Related terms

  • Goal-Conditioned Policy
  • Reinforcement Learning
  • Soft Actor-Critic
  • Markov Decision Process (MDP)
  • Curriculum Learning
  • Play Data

Related projects

  • LeRobot: HuggingFace ML library for real-world robotics in PyTorch
  • MuJoCo: DeepMind physics engine for robotics and ML research
  • PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
  • Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
  • Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
  • NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills

Videos

  • Hindsight Experience Replay | Two Minute Papers #192: Two Minute Papers

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.