MultiSCOPE: Disambiguating In-Hand Object Poses with Proprioception and Tactile Feedback

Authors: Andrea Sipos, Nima Fazeli · RSS 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Two arms tap their grasped objects together several times; particle filters scored on contact, force and cross-action consistency recover both in-hand poses to about 0.5 cm.

Why it matters

The method needs no camera and no high-resolution tactile skin, only the joint sensing and wrench estimation that cobots such as the Franka Emika Panda already provide, or an off-the-shelf wrist force-torque sensor. That makes in-hand pose correction available in occluded settings on common hardware.

Problem

Errors in the pose of a grasped object cause slip, unexpected contacts and jamming in tool use and assembly. Vision-based estimators fail under occlusion, which is common in fine assembly and narrow spaces, and high-resolution tactile sensors are costly, fatigue easily and are hard to model. A single contact (as in the authors' earlier SCOPE method) often leaves the pose ambiguous.

Proposed solution

MultiSCOPE extends SCOPE, which pairs a contact particle filter (CPFGrasp) with an object-pose particle filter, from one contact to a sequence of pokes between two objects held by two arms, using only proprioception and 6-axis wrist wrenches. It adds a tool segmentation step (K-Means on surface normals, then DBSCAN) so every face gets contact particles, a wrench loss that favours poses whose contacts explain the measured wrist wrench, and a memory loss that scores poses against a contact cloud built from earlier actions, with 'memory dropout' when an earlier action proves inconsistent.

Key insights

Combining several contact actions removes ambiguity that a single action leaves: one poke may constrain one direction, a second poke another
Sampling contact particles per segmented face, rather than uniformly over the surface, keeps small but functionally important faces from being missed and reduces the number of particles needed
Requiring estimated contacts to reproduce the measured wrist wrench (the wrench loss) was the strongest single loss in the ablation
Inter-robot calibration error matters: without calibration the authors saw 1–1.5 cm RMS error between the two arms, reduced to 3–7 mm with an ICP-based routine

Main results

Simulation, wrench tool, 10 trials with 5 actions: tool pose error 0.47 ± 0.27 cm and −0.76 ± 2.01°, probe pose error 0.49 ± 0.28 cm and −2.67 ± 4.76°; 8 of 10 tight-tolerance wrench-on-screw tasks succeeded
The full method's loss-ablation run (Table I, 5 trials) reports 0.48 ± 0.16 cm tool error and 100% task success, while the 10-trial run (text and Table II) reports 0.47 ± 0.27 cm and 80%; these are separate trial sets
Single actions alone averaged 1.08 cm tool translation error and 40% task success, vs 0.47 cm and 80% for the full multi-action method (Table II)
With force-torque noise at 5% of signal magnitude: 0.52 ± 0.31 cm tool error and 4 of 5 tasks completed; at 8% noise no task succeeded (1.13 ± 0.33 cm tool error)
Hex key, pawl and gear tools in simulation gave tool translation errors of 0.31, 0.33 and 0.49 cm (10 trials each)
Real hardware, wrench tool, 4 actions: 5 of 5 task trials succeeded (3.5 mm tolerance per side using a cylinder target)

Limitations

The authors name action selection as future work: the pokes were chosen by hand, and more principled selection is needed to reduce the number of actions
The method assumes rigid objects with known geometric models, point contacts that transmit no moment, and no relative slip during pokes
Proprioception noise is not modelled; the authors found the two real arms hard to calibrate relative to each other (3–7 mm RMS after calibration)
Accuracy degrades with force-torque noise: at 8% noise no task-based demonstration succeeded
Poses are estimated in SE(2) because the parallel-jaw grippers constrain them, and the real-world test used one tool and a cylinder instead of a screw head

Field impact

Finalist for the RSS 2023 Best Student Paper Award, with 14 citations. The paper points to www.mmintlab.com/multiscope/ for code and videos; that page links the code repository.

Abstract

In this paper, we propose a method for estimating in-hand object poses using proprioception and tactile feedback from a bimanual robotic system. Our method addresses the problem of reducing pose uncertainty through a sequence of frictional contact interactions between the grasped objects. As part of our method, we propose 1) a tool segmentation routine that facilitates contact location and object pose estimation, 2) a loss that allows reasoning over solution consistency between interactions, and 3) a loss to promote converging to object poses and contact locations that explain the external force-torque experienced by each arm. We demonstrate the efficacy of our method in a task-based demonstration both in simulation and on a real-world bimanual platform and show significant improvement in object pose estimation over single interactions. Visit www.mmintlab.com/multiscope/ for code and videos.

Open Source Robotics