RoboCrowd: Scaling Robot Data Collection through Crowdsourcing

Authors: Suvir Mirchandani, David D. Yuan, Kaylee Burns, Md Sazzad Islam, Tony Z. Zhao, Chelsea Finn, Dorsa Sadigh · ICRA 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

An ALOHA teleoperation station in a university café used candy rewards, engaging tasks and a leaderboard to collect 817 episodes from 231 people; the data helps as pretraining for ACT.

Why it matters

It tests whether members of the public, rather than designated operators, can supply useful robot demonstrations in person, and which incentives change how much and how well they demonstrate. The measured incentive effects and the quality-annotated dataset give concrete design data for anyone planning a public or shared data-collection station.

Problem

Imitation learning needs hundreds to thousands of demonstrations per task, and these are still collected by researchers or designated operators. Crowdsourcing works for text and image labels through web interfaces, but robot demonstrations need access to physical hardware, a capable robot, onboarding that novices can follow alone, and safety for untrained users.

Proposed solution

RoboCrowd places an enclosed ALOHA bimanual puppeteering setup in a public café with safety features (self-collision checking in a parallel MuJoCo simulation, an audio-visual alarm, plexiglass enclosure), ID-card sign-in, an interactive tutorial under one minute and a tablet app. Three incentive classes shape behavior: material rewards (candy delivered to an End Zone), intrinsic interest (harder tasks such as opening a Ziploc bag) and social comparison (a points leaderboard). Every episode is hand-annotated per timestep with task labels and quality scores from 0 to 3, then used to co-train or pretrain ACT policies.

Key insights

Preferred material rewards shift effort: users spent 2× as many timesteps on hi-chew as on tootsie-roll, two bin-picking tasks of near-identical difficulty (81% of 16 surveyed people preferred a Hi-Chew).
Intrinsic interest matters even with equal rewards: users spent 4.18× as many timesteps on hi-chew-ziploc as on hi-chew-bin, and 1.5× as many on jelly-bean as on hershey-kiss despite a mild preference for the Hershey Kiss.
Leaderboard visitors gave significantly more demonstrations (p<0.001) of higher mean quality (p<0.05) in a Mann-Whitney U-test.
Naively mixing crowd data with expert data can hurt, while pretraining on it and then fine-tuning on expert data more often helps.
Tutorial quality loosely predicts later task quality and can act as a first filter on demonstrators.

Main results

Over a two-week deployment (11 collection days): 231 unique users and 817 interaction episodes (129 in BinScene, 381 in Bin+DispenserScene, 307 in Bin+ZiplocScene).
Interaction time split: 54.2% on preset tasks, 9.6% on the tutorial and 36.1% on free-play.
Bin-picking success, Fine-tune versus Expert-only ACT (40 trials each): hi-chew 42.5% vs 37.5%, hershey-kiss 35% vs 20%, hi-chew-bin 40% vs 20%; tootsie-roll 40% vs 42.5%.
Long-horizon tasks (0% full success for every policy, scored by normalized return out of 100): hi-chew-ziploc 22.1 Fine-tune vs 5.4 Expert; jelly-bean 19.7 vs 48.9, where crowd data hurt.
New task tool-ziploc (50 expert demos): pretraining on crowd hi-chew-ziploc data raised staged success to 95% / 85% / 80% versus 85% / 70% / 60% for grasp corner, pinch tab and slide open.

Limitations

Crowdsourcing brings data quality and heterogeneity problems; naive co-training underperformed expert-only training on several tasks (e.g., hi-chew 27.5% vs 37.5%).
Crowd data contains meaningful but suboptimal behaviors, such as picking several candies at once, which the trained policies reproduce.
Only three incentive types were studied; the authors note open questions such as how physical rewards differ from monetary ones.
Quality labels were annotated by hand, and the long-horizon tasks reached 0% full success for all policies.
Policies were evaluated in a different location with consistent lighting, not in the café where lighting varied during collection.

Field impact

The paper was a finalist for the ICRA 2025 IEEE ICRA Best Conference Paper Award and has 11 citations. It presents its crowdsourced dataset, from over 200 users with manual fine-grained quality annotations, as a resource for future work; neither the paper nor its project page gives a code or dataset download link.

Abstract

In recent years, imitation learning from large-scale human demonstrations has emerged as a promising paradigm for training robot policies. However, the burden of collecting large quantities of human demonstrations is significant in terms of collection time and the need for access to expert operators. We introduce a new data collection paradigm, RoboCrowd, which distributes the workload by utilizing crowdsourcing principles and incentive design. RoboCrowd helps enable scalable data collection and facilitates more efficient learning of robot policies. We build RoboCrowd on top of ALOHA (Zhao et al. 2023)—a bimanual platform that supports data collection via puppeteering—to explore the design space for crowdsourcing in-person demonstrations in a public environment. We propose three classes of incentive mechanisms to appeal to users' varying sources of motivation for interacting with the system: material rewards, intrinsic interest, and social comparison. We instantiate these incentives through tasks that include physical rewards, engaging or challenging manipulations, as well as gamification elements such as a leaderboard. We conduct a large-scale, two-week field experiment in which the platform is situated in a university cafe. We observe significant engagement with the system—over 200 individuals independently volunteered to provide a total of over 800 interaction episodes. Our findings validate the proposed incentives as mechanisms for shaping users' data quantity and quality. Further, we demonstrate that the crowdsourced data can serve as useful pre-training data for policies fine-tuned on expert demonstrations—boosting performance up to 20% compared to when this data is not available. These results suggest the potential for RoboCrowd to reduce the burden of robot data collection by carefully implementing crowdsourcing and incentive design principles.

Open Source Robotics