LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models

Authors: Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, Yu Su · ICCV 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

GPT-3 with retrieved examples and object-grounded re-planning writes high-level plans for ALFRED from 100 labelled examples, nearing HLSM trained on 21,023.

Why it matters

The planner asks only for an object list and a low-level planner from the host agent and needs no parameter updates, so it can be attached to an existing hierarchical agent. Grounding comes from a pretrained object detector rather than ground-truth scene information, and commonsense in the language model replaces most of the paired language data a high-level planner normally needs.

Problem

Embodied instruction-following agents learn from large sets of paired (instruction, trajectory) data, such as thousands of ALFRED examples per task type, which is costly to collect. In a hierarchical agent only the high-level planner needs language-paired data, but it still needs a lot of it: retrained on 100 examples, the HLSM and FILM planners barely complete any task.

Proposed solution

LLM-Planner prompts GPT-3 to write a high-level plan, a sequence of subgoals such as 'Navigation fridge, OpenObject fridge', using 9 in-context examples picked from 100 training pairs by a BERT-embedding nearest-neighbour retriever, with logit biases steering output towards allowed actions and objects. When a subgoal fails or takes too many steps, it re-prompts the model with the subgoals already completed and the objects the agent's detector has seen, and HLSM's low-level planner turns each subgoal into primitive actions.

Key insights

Only the high-level planner needs language-paired data; the low-level planner can be trained on data synthesised in the simulator, so few-shot learning targets the planner alone.
Dynamic retrieval of in-context examples matters most: replacing the kNN retriever with random selection drops LOOCV high-level planning accuracy from 40.59% to 17.48%.
Adding observed objects to the prompt lets the language model map names to what is in the scene (recycle bin to garbage can, lamp to desk lamp) and guess likely object locations (a potato in the fridge, a cup in the cabinet).
Prompt format and hyperparameters were chosen by leave-one-out cross-validation on the same 100 examples, with no separate validation set (the true few-shot setting).
End-to-end failures often come from the low-level stack rather than the plan: Heat & Place reaches 36% plan accuracy but 1.8% success because HLSM's detector often misses the microwave.

Main results

With goal and step-by-step instructions, LLM-Planner + HLSM reaches 15.36% success rate on ALFRED valid unseen and 16.42% on test unseen, using 100 training pairs.
Retrained on the same 100 pairs, HLSM and FILM reach 0.00% success rate on valid unseen and 0.61% and 0.20% on test unseen.
HLSM trained on all 21,023 pairs (goal instruction only) scores 18.28% on valid unseen and 20.27% on test unseen; on test unseen, LLM-Planner's 16.42% is above full-data E.T. (8.57%), HiTUT (13.87%) and M-TRACK (16.29%).
Grounded re-planning beats static planning in every setting; with goal instructions only it adds 1.83 points of success rate on test unseen (11.58% to 13.41%).
Ablation of LOOCV plan accuracy: full model 40.59%, without kNN retriever 17.48%, without logit biases 38.10%, without both 13.43%.
Plan accuracy rises with more training examples, with diminishing returns around 250; 9 in-context examples is generally a good choice.

Limitations

The authors note a large room for improvement remains; end-to-end success is still well below the best full-data methods, such as FILM at 27.80% test unseen success rate.
Success is bounded by HLSM's low-level planner and object detector, which LLM-Planner reuses unchanged.
Plan accuracy in the dynamic setting can only be given as a range (for example 46.59–68.31% on valid unseen) because no oracle low-level controller was available.
Evaluated only in simulation (ALFRED on AI2-THOR); there are no physical robot experiments.
Depends on the GPT-3 API; other language models such as Codex, better prompt design and stronger grounding and re-planning are left to future work.

Field impact

The paper has 583 citations. It does not release code or data in the licensed version; it reuses the public GPT-3 API and the official HLSM and FILM codebases.

Abstract

This study focuses on embodied agents that can follow natural language instructions to complete complex tasks in a visually-perceived environment. Existing methods rely on a large amount of (instruction, gold trajectory) pairs to learn a good policy. The high data cost and poor sample efficiency prevents the development of versatile agents that are capable of many tasks and can learn new tasks quickly. In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models (LLMs) such as GPT-3 to do few-shot planning for embodied agents. We further propose a simple but effective way to enhance LLMs with physical grounding to generate plans that are grounded in the current environment. Experiments on the ALFRED dataset show that our method can achieve very competitive few-shot performance, even outperforming several recent baselines that are trained using the full training data despite using less than 0.5% of paired training data. Existing methods can barely complete any task successfully under the same few-shot setting. Our work opens the door for developing versatile and sample-efficient embodied agents that can quickly learn many tasks.

Open Source Robotics