Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners

Authors: Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, Anirudha Majumdar · CoRL 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

KnowNo casts LLM planning as multiple choice and applies conformal prediction, so the robot asks for help only when more than one option survives, at a user-set success level.

Why it matters

The user sets a target success level 1−ϵ, and the method guarantees task completion at that level with high probability, assuming accurate human help, while keeping the average prediction set small. It works with off-the-shelf LLMs without fine-tuning, and the coverage guarantee holds whether or not the LLM's confidences are well calibrated.

Problem

LLM planners can confidently hallucinate, and natural-language instructions are often ambiguous, so a robot that follows a wrong plan may act unsafely, for example by putting a metal bowl in a microwave. Prior LLM planners either never seek clarification or rely on extensive prompting, and none offer a way to ensure that asking for help yields a desired level of task success.

Proposed solution

The LLM first generates candidate next steps labeled A–D, an option 'E) an option not listed here' is appended, and the LLM then scores the options as a multiple-choice question through next-token probabilities. Conformal prediction, calibrated on 400 i.i.d. scenarios, sets a confidence threshold; options above it form the prediction set, and the robot executes a single remaining option or asks a human to choose otherwise. A sequence-level calibration extends the guarantee to multi-step planning, and a further extension handles steps with several acceptable options.

Key insights

Casting planning as multiple-choice Q&A removes the length bias of raw sequence probabilities; conformal prediction on raw or perplexity-normalized sentence scores gave larger prediction sets and more help in all three simulated ambiguity settings.
Conformal prediction's coverage guarantee turns into a task completion guarantee when the robot asks for help whenever the prediction set holds more than one option.
Calibrating on whole sequences, scored by the lowest per-step confidence, allows prediction sets to be built step by step at test time while keeping a guarantee over the full plan.
The guarantee makes no assumption about the LLM's calibration: with GPT-3.5, which showed a bias toward options D and E, KnowNo still met the target by triggering more human help.
An ensemble baseline can reach high success with little help, but it needs 20× the inference time and sometimes assigns zero probability to the correct option.

Main results

Across simulated and hardware settings, KnowNo met the user-specified task completion level while reducing human help by 10–24% compared with baselines.
Simulated tabletop rearrangement in PyBullet (ϵ = 0.15): KnowNo deviated least from the target success rate across attribute, numeric and spatial ambiguity, and needed up to 24% less help than Simple Set at some success levels.
Hardware multi-step sorting on a UR5 (ϵ = 0.25, 50 trials): task success 0.74, set size 1.72, step-wise help 0.58 and trial-wise help 0.92, versus 0.72, 2.04, 0.72 and 1.00 for Simple Set at the same plan success of 0.76; No Help reached 0.38 task success.
Hardware mobile manipulation in an office kitchen (ϵ = 0.15): plan success 0.87, task success 0.76, set size 2.22 and help rate 0.67, versus 0.75, 2.38 and 0.81 for Simple Set; No Help reached 0.62 plan and 0.51 task success.
Bimanual Kuka IIWA 7 setup with all-ambiguous instructions (ϵ = 0.15): 84% plan success with 92% help; on hardware the robot succeeded in 9 of 10 trials and asked for help 9 times.
Setting Simple Set to the same ϵ = 0.25 on the UR5 task made it over-conservative, with a step-wise help rate of 87%.

Limitations

Stated by the authors: the task completion guarantee assumes the environment is fully grounded in the text input to the LLM and that the actions the LLM proposes execute successfully.
Uncertainty of perception and the low-level policy is not part of the calibration; on the mobile manipulator low-level action success was about 86%, which caused gaps between plan and task success.
The method needs a labeled calibration set of i.i.d. scenarios (400 were used) drawn from the same distribution as the test scenarios.
The guarantee assumes the human gives accurate help, and the robot fails when the correct option is not in the prediction set.
GPT-3.5 (text-davinci-003) showed a strong position bias in multiple choice, which lowered performance and increased the help needed.

Field impact

Winner of the CoRL 2023 Best Student Paper Award; cited 515 times (count as given). The official project page links the code in the google-research repository, and the paper points to the prompts on the project website.

Abstract

Large language models (LLMs) exhibit a wide range of promising capabilities — from step-by-step planning to commonsense reasoning — that may provide utility for robots, but remain prone to confidently hallucinated predictions. In this work, we present KnowNo, which is a framework for measuring and aligning the uncertainty of LLM-based planners such that they know when they don't know and ask for help when needed. KnowNo builds on the theory of conformal prediction to provide statistical guarantees on task completion while minimizing human help in complex multi-step planning settings. Experiments across a variety of simulated and real robot setups that involve tasks with different modes of ambiguity (e.g., from spatial to numeric uncertainties, from human preferences to Winograd schemas) show that KnowNo performs favorably over modern baselines (which may involve ensembles or extensive prompt tuning) in terms of improving efficiency and autonomy, while providing formal assurances. KnowNo can be used with LLMs out of the box without model-finetuning, and suggests a promising lightweight approach to modeling uncertainty that can complement and scale with the growing capabilities of foundation models. Website: https://robot-help.github.io

Open Source Robotics