Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

Meta-Reinforcement Learning

  • Also called: meta-RL
  • Category: Learning & autonomy › RL & simulation
  • Tags: reinforcement-learning, meta-learning, adaptation, generalization, robot-learning

Reinforcement learning that trains on many related tasks so that the agent learns how to adapt, and can solve a new task from the same family with only a little experience.

Definition

Meta-reinforcement learning (meta-RL) treats learning itself as the thing to be learned. Instead of training a policy for one task, the agent is trained across a distribution of related tasks, for example reaching different targets, pushing objects of different masses or opening different doors. The aim is that, when given a new task drawn from the same distribution, it can adapt with a small amount of new experience, often a few episodes, rather than learning from scratch. Training is split into an inner loop, which adapts to one task, and an outer loop, which improves the adaptation procedure across tasks.

Three families of methods are common. Gradient-based methods, such as model-agnostic meta-learning (MAML, Finn, Abbeel and Levine, 2017), learn initial network weights from which a few gradient steps on the new task give a good policy. Recurrent or context-based methods, such as RL² (Duan et al., 2016), feed the history of states, actions and rewards into a recurrent network or other sequence model whose hidden state carries what it has learned about the current task, so adaptation happens without gradient updates. Latent task inference methods, such as PEARL (Rakelly et al., 2019), learn an encoder that infers a probabilistic task variable from experience and condition the policy on it, which also supports off-policy training.

Meta-RL is closely related to partially observable decision making: the unknown task can be seen as a hidden part of the state that the agent must infer by acting. It differs from multi-task reinforcement learning, which trains a single policy to perform a known, fixed set of tasks well, usually with the task identity given, and does not measure adaptation to unseen tasks. Benchmarks such as Meta-World test both settings on the same set of manipulation tasks. Results depend heavily on how broad the training task distribution is: meta-RL tends to adapt well to variations of training tasks and poorly to tasks that are structurally different.

Why it matters

Robots meet variation constantly: different objects, payloads, terrains and wear. Meta-RL is one of the main approaches to making learned controllers adapt quickly to such changes instead of being retrained from scratch, and the same ideas underlie online adaptation modules used in sim-to-real locomotion.

Key applications

  • Fast adaptation of manipulation skills to new goals and objects
  • Adapting locomotion controllers to changed payloads, terrain or damage
  • Online adaptation of learned dynamics models in model-based control
  • Benchmarking generalization with task families such as Meta-World
  • Few-shot imitation and adaptation from small amounts of robot data

In practice

A research team wants a robot arm that can learn a new manipulation skill from a handful of trials. In simulation they build a family of tasks with randomized object positions and goals across several task types, and train a context-based meta-RL agent that conditions on a latent task variable inferred from recent transitions. At test time on a held-out goal, the robot collects a few exploratory episodes, the encoder updates its estimate of the task, and the policy improves without any gradient update. The team reports success separately on held-out variations of training tasks and on entirely new task types, since performance on the latter is usually much lower.

Common misconceptions

  • Meta-RL learns a policy that can do anything — it adapts to tasks from the training distribution and usually generalizes poorly to tasks with different structure
  • Meta-RL and multi-task RL are the same — multi-task RL optimizes performance on a known set of tasks, while meta-RL optimizes the speed of adaptation to new tasks
  • Adaptation always means fine-tuning weights — context-based methods adapt purely by updating a hidden state or latent variable, with no gradient steps at test time

Who should know this

  • Robot learning researchers working on generalization and adaptation
  • Reinforcement learning practitioners designing task distributions
  • Engineers building adaptive locomotion and manipulation controllers
  • Graduate students in machine learning for robotics

Related technologies

  • Model-Agnostic Meta-Learning (MAML)
  • RL²
  • PEARL
  • Meta-World
  • Recurrent Neural Networks

Related terms

  • Multi-Task Reinforcement Learning
  • Reinforcement Learning
  • Partially Observable Markov Decision Process (POMDP)
  • Lifelong Robot Learning
  • Domain Randomization
  • Teacher-Student Training

Related projects

  • LeRobot: HuggingFace ML library for real-world robotics in PyTorch
  • MuJoCo: DeepMind physics engine for robotics and ML research
  • PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
  • OpenPI: Physical Intelligence VLA foundation models for general robot control
  • Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
  • Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim

Videos

  • Learning Exploration Strategies with Meta-Reinforcement Learning: Simons Institute for the Theory of Computing

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.