Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

Play Data

  • Also called: teleoperated play, unstructured play data
  • Category: Learning & autonomy › Learning from demonstration
  • Tags: imitation-learning, data-collection, teleoperation, goal-conditioning, robot-learning

Robot interaction data recorded while a human teleoperator freely explores an environment without a fixed task, used to train goal-conditioned and long-horizon policies.

Definition

Play data comes from a style of data collection in which a human operator teleoperates a robot, or in some work acts with their own hands, and interacts with an environment however they like rather than performing a predefined task. In a play session the operator might open a drawer, push a block, flip a switch and pick up a toy, in any order. The idea was brought into robot learning by Corey Lynch and colleagues in 2019 in their work on learning latent plans from play.

Play has several practical advantages over task demonstrations. It is cheap to collect, since there is no need to reset the scene or define success for each episode; it covers a broad range of behaviors and the transitions between them; and it naturally includes recoveries and partial attempts. Because the data has no task labels, it is usually combined with hindsight relabeling: any window of the recording can be treated as a demonstration of how to reach the state at its end, which turns unlabeled play into training data for goal-conditioned policies. Natural-language descriptions can be added afterward to a subset of windows to train language-conditioned policies.

Play data also has weaknesses. Behavior is multimodal, since the same start and goal may be connected in different ways, which pushes methods toward latent-variable or diffusion-based policies that can represent several modes. Coverage depends on the operator, so rarely visited states stay poorly learned. Some methods record humans playing with their own hands, which is faster than teleoperation, and use that data to learn high-level plans that a robot-specific policy, trained on a smaller amount of teleoperated data, then executes.

Why it matters

Collecting task-specific demonstrations for every skill does not scale. Play data lets one session of free interaction support many tasks at once, and it has become a common basis for goal-conditioned, language-conditioned and long-horizon imitation learning.

Key applications

  • Training goal-conditioned manipulation policies
  • Language-conditioned imitation learning with hindsight annotation
  • Long-horizon tasks built by chaining subgoals
  • Pretraining latent plan or skill representations
  • Multi-task tabletop manipulation benchmarks

In practice

A lab sets up a kitchen-style tabletop with a drawer, a cabinet, blocks and a light switch, and an operator teleoperates a robot arm with a VR controller for long sessions without instructions, interacting with whatever they choose. The recordings of camera images, proprioception and actions are cut into short windows during training, each labeled with its final frame as the goal. A goal-conditioned policy learns to reach any goal image, and a small subset of windows is later annotated with instructions such as "open the drawer" to enable language commands. At test time, the robot chains several goals to complete a longer task, such as opening the drawer, placing a block inside and closing it.

Common misconceptions

  • Play data is random exploration — it is purposeful human behavior; it is simply not tied to a predefined task
  • Play data needs task labels to be useful — hindsight relabeling turns any segment into a goal-reaching example without labels
  • Play data replaces task demonstrations entirely — coverage depends on the operator, so rare or precise behaviors may still need targeted demonstrations

Who should know this

  • Imitation learning researchers
  • Robot data collection and teleoperation teams
  • Engineers training language-conditioned manipulation policies
  • Researchers working on long-horizon manipulation

Related technologies

  • Hindsight Relabeling
  • Learning from Play (Play-LMP)
  • VR Teleoperation
  • CALVIN Benchmark
  • Conditional Variational Autoencoders

Related terms

  • Goal-Conditioned Policy
  • Imitation Learning
  • Teleoperation
  • Behavior Cloning
  • Visuomotor Policy

Related projects

  • LeRobot: HuggingFace ML library for real-world robotics in PyTorch
  • OpenPI: Physical Intelligence VLA foundation models for general robot control
  • NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills
  • SO-101 Open Robot Arm: 6-DOF robot arm for teleoperation and AI robotics research
  • XLeRobot: Practical dual-arm mobile home robot for $660 with <4hrs assembly
  • DimOS: Python robotics runtime with agents, SLAM and teleop, no ROS required

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.