Automated Acquisition of Structured, Semantic Models of Manipulation Activities from Human VR Demonstration

Authors: Andrei Haidu, Michael Beetz · ICRA 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

People do kitchen tasks in a physics-simulated VR world; online monitors split their actions into reaching, grasping, transport and put-down events stored in KnowRob for robots to query.

Why it matters

Demonstrations come out already segmented and annotated in an ontology the robot shares, including the virtual body parts, so no manual labelling is needed. Episodes can be replayed and queried by action type, object type and part relations, for example all grasps on handles that belong to a fridge door.

Problem

Robots learning manipulation often need more experience than can be provided with reasonable effort, so each demonstration should carry more structured information. The authors' earlier AMEvA system tracked only the head and hands, let users reach places that would be impossible in reality, could not use other body parts such as elbows or feet, and gave coarse action representations.

Proposed solution

Users act in an Unreal Engine 4 environment simulated with NVIDIA PhysX, with full-body and gaze tracking; the hands are driven by 3D PID controllers so heavy objects pull back or slip, and grasps result from physics using predefined grasp styles. A subsymbolic logger stores per-frame world states and gaze in MongoDB, while online monitors with hand contact-sensor arrays and grammars record grasp, contact, supported-by and pick-and-place sub-motions as OWL events in KnowRob.

Key insights

Driving the virtual hands through PID controllers instead of a direct mapping makes object weight matter, which prompts real-world strategies such as lifting with two hands.
Contact-sensor arrays on fingers, thumb and palm separate a real grasp from mere contact: a grasp needs contact from at least two different sets.
Motion phases without clear start or end features (start of reaching, end of pick-up, start of put-down) can be bounded with regions of interest and back-tracking over cached object motion.
Supported-by is detected as contact with near-zero relative vertical velocity, so an object carried on a tray still counts as supported by it.
Event monitors follow the physics engine's sleep state and switch off for resting objects, which the authors say lets the system scale to thousands of objects in real time.

Main results

The paper reports no quantitative accuracy or timing evaluation; its results are qualitative demonstrations and example queries.
A filled plate lifted with a one-hand lateral grasp twists out of the hand under its weight, while a two-hand grasp lifts it successfully.
Pick-and-place actions are segmented online into reaching, fixation, slide or pick-up, transport and put-down phases, shown in Fig. 5 (a)–(j).
KnowRob queries retrieve and replay sliding actions on a cereal box, grasps on handles that are part of a fridge door, and pick-and-place actions composed from sub-actions.
Per-episode MongoDB collections are indexed as B-trees once the episode ends, giving O(log n) search time.

Limitations

Full-body tracking is one-way: the avatar can push objects but gets no feedback, so a user can walk through walls or furniture and destabilise the simulation (stated by the authors).
The start of reaching and the end of pick-up have no distinct features and are approximated with regions of interest (stated by the authors).
The grasp detector does not distinguish grasp context, such as carrying an object versus holding a drawer handle or immovable furniture (stated by the authors).
Gaze is recorded but not yet used to refine action detection, and the large public dataset is still future work (stated by the authors).
No robot experiment and no quantitative evaluation of the event detection are reported.

Field impact

It was a finalist for the IEEE ICRA 2021 Best Paper Award on Human-Robot Interaction (HRI) and has 11 citations. The paper does not state a release of code or data; it names publishing a large dataset through openEASE as future work.

Abstract

In this paper we present a system capable of collecting and annotating, human performed, robot understandable, everyday activities from virtual environments. The human movements are mapped in the simulated world using off-the-shelf virtual reality devices with full body, and eye tracking capabilities. All the interactions in the virtual world are physically simulated, thus movements and their effects are closely relatable to the real world. During the activity execution, a subsymbolic data logger is recording the environment and the human gaze on a per-frame basis, enabling offline scene reproduction and replays. Coupled with the physics engine, online monitors (symbolic data loggers) are parsing (using various grammars) and recording events, actions, and their effects in the simulated world.

Open Source Robotics