Anytime, Anywhere: Human Arm Pose from Smartwatch Data for Ubiquitous Robot Control and Teleoperation

Authors: Fabian C Weigend, Shubham Sonawani, Michael Drolet, Heni Ben Amor · IROS 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

An LSTM with a 6D rotation output, a two-step calibration and Monte Carlo dropout turns one smartwatch into an arm tracker and voice-driven robot interface with ~40% lower error than prior work.

Why it matters

A consumer smartwatch costs far less than a motion capture room and needs no line of sight, so a user can stop, guide or teach a robot from wherever they stand. Users in the study stopped a robot mid-task, guided a cube by hand motion and sent it back with voice commands, and six demonstrations recorded in two minutes trained a new placing policy. The pose distribution also gives an uncertainty signal a robot controller can check before acting.

Problem

Teleoperation and programming by demonstration need human arm motion, but motion capture is expensive and stationary, Kinect-style cameras suffer line-of-sight limits, and IMU suits need two or more sensors per arm. Earlier arm-pose estimates from a single smartwatch IMU were of low accuracy. One watch reading can also match several arm postures, since human arm kinematics are redundant.

Proposed solution

A Wear OS app streams orientation, acceleration, gravity, gyroscope and air-pressure readings from a Samsung Galaxy Watch 5, paired with OptiTrack motion capture for training. A two-step calibration (watch at chest height for relative pressure, arm stretched forward for body heading) normalises the inputs. An LSTM predicts upper and lower arm rotations in a continuous 6D representation, positions follow from known arm lengths, and Monte Carlo dropout gives a distribution of poses; speech recognition on the watch microphone adds voice commands.

Key insights

Air pressure relative to a chest-height calibration correlates far better with wrist height than raw pressure (Kendall's Tau −0.308 against −0.009)
Predicting arm rotations in the continuous 6D rotation representation, then applying fixed arm lengths, beat direct Cartesian, polar and quaternion targets
LSTMs over short sequences outperformed feedforward networks, as arm motion is a time series
Monte Carlo dropout exposes multimodal pose distributions, such as a hand that could be on either side of the head, instead of hiding them in one estimate
Elbow estimates are more accurate than wrist estimates, since wrist error includes the elbow error plus forearm rotation

Main results

Best model (LSTM with 6D rotation targets): median error 2.33 cm for wrist and 1.61 cm for elbow, more than 4 cm lower in median error than results reported in two earlier single-IMU studies
RMSE 3.71 ± 2.49 cm for wrist and 2.99 ± 2.19 cm for elbow, about 40% lower than a two-IMU method reporting 6.9 ± 2.7 cm and 5.2 ± 2.6 cm
Mean combined wrist and elbow error of 2.56 cm (abstract)
Training data: 381,535 smartwatch observations at ~50 Hz from 6 participants, matched to a 12-camera OptiTrack system at ~120 Hz
150 Monte Carlo dropout predictions ran at ~40 Hz on an Intel Xeon W-2125 CPU with a GeForce RTX 2080 Ti GPU
Intervention task (3 users × 10 runs): all runs succeeded, average 22.8 ± 7.3 s from 'stop' to release, cubes placed 2.09 ± 0.97 cm from target

Limitations

The two-step calibration must be repeated whenever the user changes body orientation or location, unlike a two-IMU method that allows hip and chest movement
Fast wrist rotations or unergonomic arm motions reduce accuracy; in one case a bimodal prediction put the mean, and the avatar's arm, through the head
The user's forward-facing direction must be constant and known, and the shoulder position is assumed fixed
Training data came from 6 participants wearing the watch on the left wrist while keeping chest and hip still
Handling of multimodal predictions, such as choosing a mode by safety cost, is proposed but left to future work

Field impact

Finalist for the IROS 2023 Best RoboCup Paper Award, with 11 citations. The paper does not state a code, app or dataset release.

Abstract

This work devises an optimized machine learning approach for human arm pose estimation from a single smartwatch. Our approach results in a distribution of possible wrist and elbow positions, which allows for a measure of uncertainty and the detection of multiple possible arm posture solutions, i.e., multimodal pose distributions. Combining estimated arm postures with speech recognition, we turn the smartwatch into a ubiquitous, low-cost and versatile robot control interface. We demonstrate in two use-cases that this intuitive control interface enables users to swiftly intervene in robot behavior, to temporarily adjust their goal, or to train completely new control policies by imitation. Extensive experiments show that the approach results in a 40% reduction in prediction error over the current state-of-the-art and achieves a mean error of 2.56 cm for wrist and elbow positions.

Open Source Robotics