ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection

Authors: Arpit Bahety, Priyanka Mandikal, Ben Abbatematteo, Roberto Martín-Martín · RSS 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Models two-hand motion as a 1-DoF screw joint, extracts it from one human RGB-D video and refines it on a Tiago++ by search in screw space; 16 of 18 trials succeed across six tasks.

Why it matters

The method needs one RGB-D video per task and no teleoperation rig, motion capture or simulator. Restricting exploration to a screw axis turned three tasks that failed under 6-DoF waypoint exploration into successes within 25 real-robot episodes.

Problem

Bimanual manipulation needs two arms synchronised in space and time, which gives a large action space where random exploration on real hardware is impractical. Human videos are a cheap source of demonstrations, but hand tracking is noisy and human and robot bodies differ, so replayed motions fail and must be refined, which brings the large search space back.

Proposed solution

Following biomechanics work that models two hands as a serial kinematic chain, ScrewMimic defines a screw action: grasp points for each hand, a 1-DoF screw axis (prismatic, revolute, or revolute with fixed orientation) for the right hand's motion relative to the left, and an optional left-hand trajectory. It extracts this from one RGB-D video using a hand-object detector and FrankMocap, trains PointNet models on geometrically augmented point clouds to predict it for new object poses, and refines failing predictions with the cross-entropy method in screw-axis space using a self-generated reward from episode length and wrist force-torque readings. Successful actions are added back to retrain the predictor.

Key insights

Many two-hand tasks, with a physical joint (bottle, laptop, roll) or without one (stirring, cutting, zipping), can be described by one virtual 1-DoF screw joint between the hands.
Projecting noisy hand-tracking trajectories onto a screw axis cleans the demonstration before the robot uses it.
Exploring over screw-axis parameters instead of N×6-DoF waypoints makes real-robot fine-tuning feasible within 25 episodes.
Both reward terms are needed: without grasp-loss detection or the mean-wrench ranking, bottle opening and stirring failed.
Successful fine-tuned actions can retrain the predictor, which then succeeds almost zero-shot on the same object and adapts to new instances of the category.

Main results

New object poses (Table I): open bottle 2/3, close zipper 3/3, insert roll 3/3, close laptop 3/3, stir 2/3, cut 3/3, i.e. 16 of 18 trials, which the paper reports as 90% aggregate success.
Successful trials needed 7 to 18 episodes on average (1 to 3 epochs of 5 episodes), with a cap of 25 episodes; fine-tuning takes about 40 minutes.
Action-space ablation (Table III): screw-space exploration succeeded on bottle, roll and laptop after 16, 7 and 11 of 25 episodes; FM + N×6-DoF (DEFT-style) and Screw + N×6-DoF exploration failed all three within 25 episodes, reaching 0–50% task completion.
Retraining on the corrected action (Table II): on the same object, open bottle went from 3 epochs/16 episodes to 0/1 and insert roll from 1/7 to 0/2; on a new bottle the retrained model needed 2/14, and after a further round 0/1.
With the largest added noise (σ = 3.0 cm, 12.5°), the extracted axis was off by 2.1 cm and 13.1°, and fine-tuning still succeeded after 4 epochs and 21 episodes.
A PointNet pretrained on ModelNet-40 reduced axis error on unseen bottles (e.g. from 0.09 m, 6° to 0.03 m, 4.2°) while error on the training bottle stayed the same.

Limitations

A single screw axis does not fit tasks where the hands move along more than one axis, such as cutting in a zig-zag.
A separate prediction model is trained for each object class, which limits generalisation.
Episode success is recorded manually, and a human resets the scene when needed.
Only the screw axis is fine-tuned, not the grasp contact points; reflective surfaces such as the laptop had to be covered because the depth sensor could not see them from all angles.
With at most 25 episodes per trial, results depend on the first screw-action samples; the two failed trials ended without finding a successful action.

Field impact

Finalist for the RSS 2024 Outstanding Student Paper Award, with 44 citations. The official project page links the code at github.com/UT-Austin-RobIn/ScrewMimic.

Abstract

Bimanual manipulation is a longstanding challenge in robotics due to the large number of degrees of freedom and the strict spatial and temporal synchronization required to generate meaningful behavior. Humans learn bimanual manipulation skills by watching other humans and by refining their abilities through play. In this work, we aim to enable robots to learn bimanual manipulation behaviors from human video demonstrations and fine-tune them through interaction. Inspired by seminal work in psychology and biomechanics, we propose modeling the interaction between two hands as a serial kinematic linkage — as a screw motion, in particular, that we use to define a new action space for bimanual manipulation: screw actions. We introduce ScrewMimic, a framework that leverages this novel action representation to facilitate learning from human demonstration and self-supervised policy fine-tuning. Our experiments demonstrate that ScrewMimic is able to learn several complex bimanual behaviors from a single human video demonstration, and that it outperforms baselines that interpret demonstrations and fine-tune directly in the original space of motion of both arms. For more information and video results, https://robin-lab.cs.utexas.edu/ScrewMimic/

Open Source Robotics