On Evaluation of Embodied Navigation Agents

Authors: Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, Amir R. Zamir · arXiv 2018

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence.

TL;DR

A working group's recommendations report on navigation research, not a new method: a DONE action, geodesic distance, the SPL metric, continuous SI-unit simulators and standard scenarios on four datasets.

Why it matters

SPL gives one number that rewards both reaching the goal and taking an efficient route, and the DONE requirement tests whether the agent knows it has arrived. The paper also asks that simulation work state its sensing assumptions, such as perfect odometry or GPS, and that simulators ship with open-source software for deploying trained agents on physical robots.

Problem

Navigation research in 3D environments had produced many incompatible task definitions, testbeds and evaluation protocols, which makes results hard to compare. Some protocols counted an episode as successful as soon as the agent came near the goal, measured closeness with Euclidean distance that ignores walls, or trained in the test environment without reporting how much exposure the agent had.

Proposed solution

A working group of navigation researchers defines goal types (PointGoal, ObjectGoal, AreaGoal), regimes of prior exposure to the test environment, and seven recommendations. The primary measure is SPL: each successful episode scores the shortest-path length divided by the larger of the agent's path length and the shortest path, averaged over all episodes, and success is judged only when the agent issues a DONE action within a distance threshold of the goal. Standard training, validation and test scenarios are defined for SUNCG, Matterport3D, AI2-THOR and Gibson.

Key insights

Success should be judged at the moment the agent declares DONE, not at some more favourable earlier moment in the episode.
Geodesic (shortest-path) distance should replace Euclidean distance, since an agent next to the goal but behind a wall is not close.
SPL is stringent: the authors expect 0.5 to be a good level in complex unseen environments and suggest calibrating it per dataset with human subjects.
Prior exposure to the test environment must be quantified and reported; the paper names three regimes: no prior exploration, pre-recorded exploration, and time-limited exploration by the agent.
Continuous state spaces and SI units (1 simulator unit = 1 m) bring simulators closer to deployment conditions and make results comparable.

Main results

SPL = (1/N) Σ Sᵢ · ℓᵢ / max(pᵢ, ℓᵢ): 50% success on optimal paths gives 0.5, 100% success on paths twice the optimum gives 0.5, and 50% success at twice the optimum gives 0.25.
Default success threshold τ is twice the agent's body width; with the 0.2 m body width used for SUNCG and Matterport3D, and as Gibson's suggested default, τ = 0.4 m.
SUNCG scenarios: 500 single-floor houses split 300/100/100, with 2,737 rooms, 41,158 objects and about 110,000 m² of floor; Matterport3D: the dataset's 61/11/18 house split, 2,206 room regions on 190 floors.
AI2-THOR v1.0: 120 scenes in 4 room types, 80 for training, 20 for validation and 20 for testing, with 125 scenarios per test scene for PointGoal and for ObjectGoal.
Gibson: 572 buildings, 1,447 floors and 211,000 m², with tiny (25/5/5), medium (100/20/20) and full (402/85/85) partitions.
Scenario counts: the general procedure in Section 6 generates more than 100 start-goal configurations per scene, while the Gibson subsection specifies 100 start-goal pairs per space.

Limitations

This is a recommendations report from a working group, not an experimental study: it runs no experiments with robots, simulated or real, and its results are definitions, a metric and standard scenarios.
The authors note that simulation is regarded with scepticism in robotics and that agents must eventually work in the physical world; tools for transfer to real robots are a recommendation, not something the paper provides.
The group declines to recommend any particular internal representation for navigation agents, calling it too early.
AI2-THOR v1.0 cannot be used for AreaGoal because each scene consists of a single room.
Gibson geodesic distances are computed at a fixed height above the floor and differ by agent type, such as a ground vehicle versus a drone, so they may need recomputing per agent.

Field impact

Cited 1047 times. The paper publishes standard navigation scenarios: SUNCG and Matterport3D scenario files at github.com/minosworld/scenarios and Gibson scenarios at gibson.vision/nav_scenarios.

Abstract

Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, present evaluation measures, and provide standard scenarios that can be used for benchmarking.

Open Source Robotics