CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Painting

Authors: Peter Schaldenbrand, Gaurav Parmar, Jun-Yan Zhu, James McCann, Jean Oh · ICRA 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Fine-tuning a text-to-image model on a painting robot's simulated strokes teaches it what the robot can paint and how to extend human work, cutting the semantic sim-to-real gap from 0.241 to 0.035.

Why it matters

Large pre-trained image models know nothing about a specific robot's tools, which limits their use for physical planning in general. The paper shows that fine-tuning on the robot's own simulated output encodes those constraints, and the open-source code is available for XArm, Franka Emika and Rethink Sawyer robot platforms.

Problem

FRIDA, an earlier robot painting system, lets users contribute only an initial prompt or image, and its CLIP-guided planning produces noisy paintings that only loosely match the text. Pre-trained text-to-image models give clearer images, but they do not know the robot's brush, colour and stroke limits, and when asked to edit a canvas they often make large changes that overwrite the human's existing work.

Proposed solution

CoFRIDA uses FRIDA's simulator to paint images from the art subset of LAION, then removes strokes (all of them, a random subset, a salient region or a semantic region) to make partial and full painting pairs, discarding pairs whose simulated painting has CLIPScore below 0.5 against its caption. Instruct-Pix2Pix is fine-tuned at a low learning rate to predict the full painting from the partial one and the text; at run time it proposes how the canvas should look, FRIDA plans the strokes to get there, and the human and robot take turns until the user is satisfied.

Key insights

Co-painting differs from in-painting: the edit must preserve and engage with the whole canvas rather than re-imagine a user-marked local region.
Besides the pixel-level sim-to-real gap there is a semantic one: an image the robot cannot render loses meaning when painted, measured here as the CLIP-embedding cosine distance between the generated image and its simulated painting.
After fine-tuning, the model adapts to each setting: prominent content for a large brush, few colours for a fixed palette or marker, sparse drawings under a stroke limit.
Planning semantics in pixel space with a pre-trained model and then handing off to a low-level stroke planner avoids the noisy output of optimising strokes directly against a text goal.

Main results

On 40 co-painting cases per method (Sharpie drawings started from 35 simulated strokes), the semantic gap ∆sem fell from 0.241 without fine-tuning to 0.035 with it, and the pixel gap ∆pix from 0.195 to 0.052.
CLIPScore was 0.741 for FRIDA, 0.624 for CoFRIDA and 0.595 without fine-tuning (BLIPScore 0.192, 0.178, 0.162); FRIDA optimises CLIPScore directly, so the authors treat these scores as favouring it.
In an MTurk survey of 24 participants (4 ratings per image pair, 160 comparisons per baseline), CoFRIDA was generally chosen as matching the caption more clearly than FRIDA and than CoFRIDA without fine-tuning, though many participants found neither image fit.
Across the LAION fine-tuning data, ∆pix had an insignificant correlation with CLIPScore (Pearson −0.08, p = 0.08) while ∆sem had a significant negative one (−0.48, p = 2.4e−31), which the authors read as the model learning robot abilities rather than a style transfer.
Real paintings with acrylics and a Sharpie engaged with human content outside the fine-tuning data, including watercolour, even though fine-tuning used a single medium.

Limitations

Limited to discrete turn-taking interaction between human and robot.
Partial paintings for training are made by hand-designed stroke removal; an end-to-end approach that learns how to form them could do better.
Subject to the biases of Stable Diffusion and its training data, so the authors recommend research use only; after harmful content was reported in LAION they switched the code to COCO without seeing a drop in quality.
Sim-to-real gaps are measured against FRIDA's simulation rather than real paintings, because generating many real samples was infeasible.
The robot can use only one painting setting (12-colour acrylic, fixed 4-colour acrylic, or black Sharpie) at a time.

Field impact

Winner of the IEEE ICRA 2024 Best Paper Award on Human-Robot Interaction, with 32 citations. The code is released as open source in the FRIDA repository and is available for XArm, Franka Emika and Rethink Sawyer robot platforms.

Abstract

Prior robot painting and drawing work, such as FRIDA, has focused on decreasing the sim-to-real gap and expanding input modalities for users, but the interaction with these systems generally exists only in the input stages. To support interactive, human-robot collaborative painting, we introduce the Collaborative FRIDA (CoFRIDA) robot painting framework, which can co-paint by modifying and engaging with content already painted by a human collaborator. To improve text-image alignment–FRIDA’s major weakness–our system uses pre-trained text-to-image models; however, pre-trained models in the context of real-world co-painting do not perform well because they (1) do not understand the constraints and abilities of the robot and (2) cannot perform co-painting without making unrealistic edits to the canvas and overwriting content. We propose a self-supervised fine-tuning procedure that can tackle both issues, allowing the use of pre-trained state-of-the-art text-image alignment models with robots to enable co-painting in the physical world. Our open-source approach, CoFRIDA, creates paintings and drawings that match the input text prompt more clearly than FRIDA, both from a blank canvas and one with human created work. More generally, our fine-tuning procedure successfully encodes the robot’s constraints and abilities into a foundation model, showcasing promising results as an effective method for reducing sim-to-real gaps.

Open Source Robotics