Learning on the Job: Long-Term Behavioural Adaptation in Human-Robot Interactions

Authors: Francesco Del Duchetto, Marc Hanheide · IROS 2022

Licences

Open Access
Paper: CC BY-SA 4.0

Everything below draws on the full paper under its CC BY-SA 4.0 licence; its figure is credited to the authors.

TL;DR

Lindsey, a museum tour-guide robot, adapts its tours online with UCBVI using camera-estimated visitor engagement as reward, raising exhibits visited by 22.8% and completion by ~30%.

Why it matters

The learning runs on a real robot in an unstructured public space, where untrained visitors can interrupt or walk away at any moment, rather than in a lab or simulation. It uses an automatically estimated engagement signal as the only feedback, with no separate data-collection and training phases. A two-week return to the static policy after learning checks that the gains came from the learned policy and not from other effects such as post-lockdown visitor behaviour.

Problem

Lindsey had given more than 2,300 tours in 103 days at The Collection museum in Lincoln, UK, but only about 18% were completed by visitors. A social robot must learn from experience how its actions affect users, yet the objective, user engagement, is an internal state that can only be estimated from sensors. Exploration on a real robot is also costly in time and limited by the dynamics of the real world.

Proposed solution

Tours are specified as Petri net plans whose actions (go to an exhibit, describe it, describe it in more detail, end the tour) are constrained to valid successor sequences. An end-to-end regression model on the robot's GPU estimates engagement from the camera at about 1 Hz; its average over each action is binned into LOW, MEDIUM or HIGH and used both in the state and as the reward. The Upper-Confidence-Bound Value Iteration (UCBVI) algorithm, with its visitation counts initialised from thousands of tours under the original static policy and an exploration bonus set by σ = 20, updates the policy after every tour.

Key insights

Visitors showing LOW engagement at the first stop have about a 20% lower chance of continuing the tour, which motivates using engagement to choose actions.
The learned policy reorders exhibits and decides when to give additional descriptions: it gives them more often early in the tour, and later only when engagement is MEDIUM or HIGH.
Seeding UCBVI with static-policy tours reuses that data while the bonus directs exploration to less-explored state–action pairs; the bonus also makes the robot repeat a poorly explored action over consecutive tours before trying a new one, which the authors found useful for consistent behaviour.
Because the reward is always positive, the reward function implicitly favours longer tours.

Main results

During the 8-week learning phase the average number of exhibits visited per tour rose 22.8% over the previous static-policy period, and returned to the earlier level in the 2-week verification phase with the static policy.
The tour success rate (tours that visit all stops) rose by over 30% compared with the static tour; the abstract states this as a 30% increase in the probability of completing the tour.
Under the learned policy, engagement stayed stable between consecutive actions almost 50% of the time, against roughly equal chances of increasing, decreasing or staying stable before.
Exploration reached only about 13% of the state–action space by the beginning of 2022; a simulated 5-week continuation showed the number of newly explorable states decreasing over time.
Average detected engagement decreased during the learning phase compared with the earlier and verification periods.

Limitations

Only a fraction of the state–action space was explored, and the authors state complete exploration cannot be practically achieved in this real-world setting; some states can only be reached when user behaviour allows it.
Detected engagement dropped after the museum reopened from COVID lockdown, possibly because mask wearing affects the engagement model; the study spans almost three years with many factors influencing visitor behaviour.
The UCBVI bonus cannot recognise when estimates for well-explored areas become outdated because user preferences shift; the authors propose a bonus based on divergence from past estimates as future work.
Apart from an overall decrease in disengagement, the authors find no other significant difference in tour stopping or abandonment between the learned and static policies.
The study covers one robot in one museum gallery with four tours of five or six exhibits, and only guided-tour interactions are considered.

Field impact

The paper was a finalist for the IROS 2022 Best Paper Award on Cognitive Robotics (sponsored by KROS) and has 14 citations. The paper does not release code or data.

Abstract

In this work, we propose a framework for allowing autonomous robots deployed for extended periods of time in public spaces to adapt their own behaviour online from user interactions. The robot behaviour planning is embedded in a Reinforcement Learning (RL) framework, where the objective is maximising the level of overall user engagement during the interactions. We use the Upper-Confidence-Bound Value-Iteration (UCBVI) algorithm, which gives a helpful way of managing the exploration-exploitation trade-off for real-time interactions. An engagement model trained end-to-end generates the reward function in real-time during policy execution. We test this approach in a public museum in Lincoln (UK), where the robot is deployed as a tour guide for the visitors. Results show that after a couple of months of exploration, the robot policy learned to maintain the engagement of users for longer, with an increase of 22.8% over the initial static policy in the number of items visited during the tour and a 30% increase in the probability of completing the tour. This work is a promising step toward behavioural adaptation in long-term scenarios for robotics applications in social settings.

Open Source Robotics