RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools

Authors: Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, Jiajun Wu · CoRL 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

RoboCook models dough-tool interaction with GNNs trained on 20 minutes of real data per tool, picks tools with a PointNet++ classifier and makes dumplings using 15 interchangeable tools.

Why it matters

The system shows a general-purpose arm with a tool-changing setup completing a nine-step dumpling sequence and recovering when a person deforms, adds to or replaces the dough mid-task. Because the policy needs one forward pass per action, planning takes seconds rather than the minutes that optimization-based planners need with the same learned model.

Problem

Long-horizon soft-body tasks such as making a dumpling need a discrete choice of which tool to use and a continuous choice of how to move it, while the elasto-plastic dough deforms in ways that are hard to model. Prior learned-dynamics methods handled short-horizon tasks with a single tool, and simulators for elasto-plastic objects show a noticeable sim-to-real gap.

Proposed solution

Four RGB-D cameras give a merged point cloud from which the dough is reconstructed as a watertight mesh and resampled to 300 surface particles, concatenated with particles on the tool mesh. A GNN trained on about 20 minutes of random-exploration data per tool predicts how the dough deforms; a PointNet++ classifier proposes the three most likely tools for the next subgoal; and a PointNet++ policy, trained on synthetic data generated by the GNN, outputs action parameters for gripping, pressing and rolling, while the other tools use hand-coded planners. The loop closes with visual feedback after each action.

Key insights

Training the GNN to predict only two steps ahead was enough for accurate 15-step predictions at inference time.
A policy trained offline on synthetic GNN rollouts covers the whole action space and outperformed online planners (gradient descent, CEM and RL) that used the same GNNs and tool classifier.
Per-tool action spaces of two to four parameters, derived from the 6-DoF end-effector space by removing redundant degrees of freedom, keep planning tractable.
A dynamics model trained only on flour, water and salt dough shaped a letter K in Play-Doh, air-dry clay and model foam without retraining.
A PointNet-based tool classifier generalized better than a ResNet-based one, which the authors attribute to its ability to encode depth.

Main results

Letter shaping (R, O, B, C, K): 100 human subjects recognized RoboCook's letters with accuracy 0.90 ± 0.11, versus 0.54 (RoboCraft), 0.52 (CEM+GNN), 0.48 (CEM+MPM) and 0.17 (RL+GNN).
Point-cloud metrics: Chamfer distance 0.0062 ± 0.0007 and Earth Mover's distance 0.0042 ± 0.0006, the lowest among the five methods.
Planning time 9.3 ± 1.5 s versus 613.7 s (RoboCraft), 756.0 s (CEM+GNN), 1486.7 s (CEM+MPM) and 1867.9 s (RL+GNN); the complete dumpling pipeline takes around 10 s of planning.
The tool classifier reached 0.996 instance accuracy on a test split of its training data.
Five human subjects using the same tools did notably worse than RoboCook when choosing tools themselves, and comparable or worse when given the tool sequence and subgoals.

Limitations

Stated by the authors: the dough occasionally sticks to the tool, and an automatic error-correction system would be needed.
Stated by the authors: the system relies on human priors for each tool's action space.
Stated by the authors: humans define the subgoals; higher-level temporal abstraction and task-level planning would be needed to remove them.
Stated by the authors: cables and cloth would need additional topology estimation.
Learned dynamics and policies cover only grippers, rollers, presses and punches; the knife, cutter, pusher and spatulas use hand-coded planners, and the hook action is pre-coded.

Field impact

Winner of the CoRL 2023 Best Systems Paper Award; cited 137 times (count as given). The official project page links the RoboCook code repository and a data folder.

Abstract

Humans excel in complex long-horizon soft body manipulation tasks via flexible tool use: bread baking requires a knife to slice the dough and a rolling pin to flatten it. Often regarded as a hallmark of human cognition, tool use in autonomous robots remains limited due to challenges in understanding tool-object interactions. Here we develop an intelligent robotic system, RoboCook, which perceives, models, and manipulates elasto-plastic objects with various tools. RoboCook uses point cloud scene representations, models tool-object interactions with Graph Neural Networks (GNNs), and combines tool classification with self-supervised policy learning to devise manipulation plans. We demonstrate that from just 20 minutes of real-world interaction data per tool, a general-purpose robot arm can learn complex long-horizon soft object manipulation tasks, such as making dumplings and alphabet letter cookies. Extensive evaluations show that RoboCook substantially outperforms state-of-the-art approaches, exhibits robustness against severe external disturbances, and demonstrates adaptability to different materials.

Open Source Robotics