FRIDA: A Collaborative Robot Painter with a Differentiable, Real2Sim2Real Planning Environment

Authors: Peter Schaldenbrand, James McCann, Jean Oh · ICRA 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A robot painter that plans brush strokes in a differentiable simulator built from its own real strokes, then replans from camera images to meet text, image or sketch goals.

Why it matters

The system joins content generation from pretrained vision models with action planning for a physical arm, and it adapts to how real paint and brushes behave rather than assuming perfect execution. Because the simulator is fitted to a small set of the robot's own strokes, it can be recalibrated when the brush or materials change, and the authors state that any ROS-compatible arm with a morphology similar to the Sawyer could be adapted with minimal interface changes.

Problem

Painting is a dynamic process with high-level, semantic goals, but most robot painting systems treat it as printing: a full stroke plan is made in advance and executed blindly. Learning-enabled painting systems plan in simulators with a large sim2real gap, so their stroke plans give poor paintings or cannot be executed on a real robot.

Proposed solution

During calibration the robot paints random strokes, and a small network (param2stroke) learns to map three stroke-shape parameters (thickness, bend, length) to the stroke's appearance; with differentiable placement and stamping onto the canvas this forms a fully differentiable painting simulator. A plan of brush strokes is optimised by gradient descent against CLIP text, VGG style, pixel or CLIP-feature losses, and after each batch of executed strokes the camera image of the real canvas is used to re-optimise the remaining plan.

Key insights

Fitting the stroke model to strokes the robot actually painted (real2sim2real) gives a smaller sim2real gap than the unconstrained or hand-constrained stroke models of earlier learning-based painters.
A fully differentiable rendering pipeline lets losses from pretrained networks such as CLIP and VGG drive brush-stroke parameters directly through gradient descent.
Re-optimising the remaining plan from camera images after each batch of strokes lets the plan adapt to stochastic execution; without replanning, deviation from the initial plan grows linearly stroke by stroke.
A semantic replication loss on CLIP convolutional features captured target content better than a pixel L2 loss in a crowd survey.
Ordering strokes from light to dark reduces brush-cleaning time and stops the brush dragging wet dark paint into light regions.

Main results

Average L1 distance between modelled and real strokes over 50 samples: 0.012 (training strokes) and 0.018 (test strokes) for param2stroke vs 0.028 and 0.028 for DiffVG; the difference is significant at p < .01.
Sim2real gap on a portrait, measured as MSE between the simulated plan and the real painting: 0.029 for FRIDA vs 0.066 for Schaldenbrand & Oh 2021 and 0.156 for Huang et al. 2019.
In a survey of 103 Amazon Mechanical Turk participants, 73.3% (replication question) and 68.3% (high-level content question) preferred paintings made with the Semantic Replication Objective; both are significantly above 50% at p = 0.01.
Stroke-model error falls steadily as more calibration strokes are used, with small gains after about 100 training/validation strokes (tested on 42 held-out strokes).
Demonstrated tasks on the real robot include painting from a language description in the style of an example image, painting a target image conceptually, and painting from a user sketch.

Limitations

The painting model is simplified: discrete colour options, strokes assumed independent, no modelling of paint wetness or of how much paint is on the brush, and the brush is fixed perpendicular to the canvas.
The system's notion of a dynamic creative process is far more constrained than the one it draws on; it does not yet explore or create new styles.
No automatic paint mixing: a human mixes and supplies the discrete palette, and the positions of canvas, paints, water and rag are explicitly programmed.
The amount of paint on the brush is the largest unmodelled variable and adds noise to the stroke model.
Setup needs indirect, diffused lighting to avoid glare from wet paint, and calibration needs a human to mark the canvas corners and set the brush's lightest and hardest touch.

Field impact

Finalist for the ICRA 2023 Outstanding Deployed Systems Paper award, with 56 citations. The complete FRIDA software stack is released as open source.

Abstract

Painting is an artistic process of rendering visual content that achieves the high-level communication goals of an artist that may change dynamically throughout the creative process. In this paper, we present a Framework and Robotics Initiative for Developing Arts (FRIDA) that enables humans to produce paintings on canvases by collaborating with a painter robot using simple inputs such as language descriptions or images. FRIDA introduces several technical innovations for computationally modeling a creative painting process. First, we develop a fully differentiable simulation environment for painting, adopting the idea of real to simulation to real (real2sim2real). We show that our proposed simulated painting environment is higher fidelity to reality than existing simulation environments used for robot painting. Second, to model the evolving dynamics of a creative process, we develop a planning approach that can continuously optimize the painting plan based on the evolving canvas with respect to the high-level goals. In contrast to existing approaches where the content generation process and action planning are performed independently and sequentially, FRIDA adapts to the stochastic nature of using paint and a brush by continually re-planning and re-assessing its semantic goals based on its visual perception of the painting progress. We describe the details on the technical approach as well as the system integration. FRIDA software is freely available at: https://pschaldenbrand.github.io/frida/.

Open Source Robotics