Goal-Conditioned Policy
- Also called: goal-conditioned RL, goal-conditioned reinforcement learning, GCRL, goal-conditioned imitation learning
- Category: Learning & autonomy › RL & simulation
- Tags: reinforcement-learning, imitation-learning, robot-learning, navigation, multi-task-learning
A control policy that takes a goal as an extra input alongside its observations, so one learned model can reach many different targets instead of being trained for a single task.
Definition
Goal-conditioned policies choose actions based on both the current observation and a specified goal, written π(a | s, g). A standard policy learns to do one thing; a goal-conditioned policy learns a family of behaviors indexed by the goal. The same network can drive a robot to many locations, place an object in many poses, or reach any target image it is given.
Goals can take many forms: a target position or pose, a desired joint configuration, an image of what the scene should look like, or a language instruction. In navigation, image goals are common, where the robot is shown a photo of a place and must reach it. In manipulation, goals are often target object poses or images of the desired arrangement.
In reinforcement learning, goal-conditioning is formalized through goal-conditioned value functions, such as universal value function approximators, which estimate how well the robot can reach each goal from each state. A practical difficulty is that rewards for reaching a specific goal are rare. Hindsight experience replay addresses this by relabeling a failed attempt as a success for the goal that was actually reached, turning every trajectory into useful training data.
Goal-conditioning is also central to imitation learning from large datasets. Demonstrations or recorded play can be relabeled with goals taken from later in the same trajectory, so a single dataset teaches the policy how to reach many outcomes. Goal-conditioned policies are often combined with planners or maps that supply a sequence of intermediate goals, letting a short-horizon policy accomplish long-horizon tasks.
Why it matters
Goal-conditioned policies let one model serve many tasks, which makes better use of expensive robot data and gives planners a reusable low-level skill to call. They are a common interface between high-level task specification and learned control.
Key applications
- Image-goal and position-goal navigation
- Multi-task manipulation where the target object pose or arrangement varies
- Reaching and pushing tasks trained with hindsight relabeling
- Low-level skills called by planners in hierarchical systems
In practice
A navigation team might train a policy that takes the current camera image and an image of the target location and outputs velocity commands. Training data comes from logged robot traversals, where later frames from the same run are used as goals for earlier frames. At deployment a topological map or planner supplies a sequence of subgoal images, and the policy drives toward each one in turn. Teams measure success across held-out goals rather than only the goals seen in training.
Common misconceptions
- A goal-conditioned policy can reach any goal — it generalizes only across goals similar to those represented in training
- Goal-conditioning only applies to reinforcement learning — it is equally common in imitation learning, where goals are relabeled from demonstration data
- A goal-conditioned policy replaces planning — for long tasks it usually needs a planner or map to provide nearby subgoals
Who should know this
- Robot learning researchers working on reinforcement and imitation learning
- Navigation engineers using learned controllers
- Engineers building hierarchical planning and control systems
Related technologies
- Reinforcement Learning
- Hindsight Experience Replay
- Universal Value Function Approximators
- Imitation Learning
- Diffusion Policy
- Topological Map
Related terms
Related projects
- LeRobot: HuggingFace ML library for real-world robotics in PyTorch
- MuJoCo: DeepMind physics engine for robotics and ML research
- PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
- OpenPI: Physical Intelligence VLA foundation models for general robot control
- Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
- Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim