FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding

Authors: Huy Ha, Shuran Song · CoRL 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A dual-arm robot learns, self-supervised, where to grasp a crumpled cloth for a fixed pick-stretch-fling motion, beating quasi-static pick & place on coverage and step count.

Why it matters

The paper shows that a single learned high-velocity primitive can unfold cloth in a few actions, handle cloth larger than the reach range and transfer from rectangles to T-shirts. The real system detects a stretched cloth with a second RGB-D camera instead of force sensors, and the training signal comes from a top-down camera mask, which keeps the hardware simple.

Problem

Most cloth-unfolding systems use single-arm quasi-static actions such as pick & place, which need many interactions on badly crumpled cloth and cannot handle cloth larger than the arm's reach range. Heuristic methods based on corners, edges or wrinkles fail when those keypoints are occluded, not visible, or absent on non-square cloth.

Proposed solution

FlingBot fixes a dual-arm motion primitive (top-down pinch grasp at two points, lift to 0.30 m and stretch taut, fling forward 0.70 m at 1.4 m/s, pull back 0.50 m, place) and learns only where to grasp, parameterised by a centre pixel, an angle and a width so that the arm-ordering and grasp-width constraints stay linear and independent. A value network predicts per-pixel delta-coverage on a top-down image at 12 rotations and 8 scales (spatial action maps); it is trained self-supervised in a custom PyFleX-based simulator and then fine-tuned on two real UR5 arms.

Key insights

Fixing the stretch and fling motion reduces learning to choosing two grasp points; also learning the fling speed (FlingBot-S) gave no significant gain.
Parameterising the two-arm grasp by centre, angle in [−90°, 90°] and width makes the left-of-right and arm-distance constraints linear and independent.
Using translation, rotation and scale equivariance through spatial action maps is essential: direct regression of the grasp parameters (Fling-Reg) reached only +0.3% delta coverage on normal rectangles.
Through trial and error the policy learned to grasp visible corners and edges, or to fling to reveal them, a strategy earlier heuristic methods designed by hand.
Flinging expands the effective reach range, so cloth larger than the arms' reach can still be unfolded.

Main results

Simulation, normal rectangles (final / delta coverage): FlingBot 91.9% / +63.1% against Pick&Place 53.0 / +24.2, Pick&Drag 58.0 / +29.2 and Stretch&Drag 77.2 / +48.4.
Over 80% coverage within 3 actions on novel normal rectangular cloths in simulation.
Large rectangles beyond reach range: 79.2% final and +52.0% delta coverage, about twice the quasi-static baselines (+24.8%, +27.1%, +23.1%).
T-shirts, unseen in training: 93.3% final / +46.8% delta coverage in simulation.
Real dual-UR5 system, 10 test episodes per cloth type: final coverage 81.9%, 88.5% and 89.2% on normal rectangle, large rectangle and shirt, against 43.2%, 38.4% and 42.7% for Pick&Place; fine-tuning raised normal-cloth coverage from 69.8% to 81.9%.
Median time per fling primitive 16.7 s against 8.8 s for pick & place, plus 2.5 s and 2.9 s overheads for both.

Limitations

Future work is needed for more complex, goal-conditioned cloth tasks such as folding, which the authors suggest addressing by combining dynamic and quasi-static actions with cloth pose estimation.
Experiments use a plain black workspace and non-textured coloured cloth for easy masking; arbitrary textures would need added segmentation or background subtraction.
Real-world results are averaged over 10 test episodes per cloth type with real-world grasp errors filtered out.
A fling takes longer than a pick & place (median 16.7 s against 8.8 s), even though it needs fewer actions.
The real setup relied on a 2-inch rubber mat and rubber gripper finger covers to help pinch grasps, and training to convergence took about 6 days on a GTX 1080 Ti.

Field impact

Winner of the CoRL 2021 Best System Paper Award, with 229 citations as given. The authors release code, the simulation environment, data and experiment videos via the project page, including a custom simulator on PyFleX that loads arbitrary cloth meshes.

Abstract

High-velocity dynamic actions (e.g., fling or throw) play a crucial role in our everyday interaction with deformable objects by improving our efficiency and effectively expanding our physical reach range. Yet, most prior works have tackled cloth manipulation using exclusively single-arm quasi-static actions, which requires a large number of interactions for challenging initial cloth configurations and strictly limits the maximum cloth size by the robot's reach range. In this work, we demonstrate the effectiveness of dynamic flinging actions for cloth unfolding with our proposed self-supervised learning framework, FlingBot. Our approach learns how to unfold a piece of fabric from arbitrary initial configurations using a pick, stretch, and fling primitive for a dual-arm setup from visual observations. The final system achieves over 80% coverage within 3 actions on novel cloths, can unfold cloths larger than the system's reach range, and generalizes to T-shirts despite being trained on only rectangular cloths. We also finetuned FlingBot on a real-world dual-arm robot platform, where it increased the cloth coverage over 4 times more than the quasi-static baseline did. The simplicity of FlingBot combined with its superior performance over quasi-static baselines demonstrates the effectiveness of dynamic actions for deformable object manipulation.

Open Source Robotics