Spatial Assisted Human-Drone Collaborative Navigation and Interaction through Immersive Mixed Reality

Authors: Luca Morando, Giuseppe Loianno · ICRA 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Mixed-reality framework where a HoloLens 2 user and a quadrotor merge their maps, and the user steers the drone with gesture forces kept safe by RRT* planning and obstacle repulsion.

Why it matters

Drones for inspection, warehouses and search and rescue often fly near obstacles the operator cannot see directly. Showing the operator what the drone maps, and combining the operator's gesture force with a planned path or an obstacle repulsion field, gives a way to share control between full teleoperation and full autonomy.

Problem

Human-drone collaboration is still mostly one-way teleoperation with joysticks, keyboards or pointers and feedback on flat monitors. Earlier extended-reality drone interfaces show the drone in a virtual scene built in advance, or let the user send goal positions or draw trajectories, rather than giving continuous spatial interaction tied to what the drone itself perceives.

Proposed solution

A Spatial Awareness Module fuses the drone's stereo point cloud and the headset's point cloud into one Voxblox map, shown to the user as egocentric through-the-wall meshes and as an exocentric merged map. The user's pinch-and-drag gesture becomes a virtual force fed to an admittance controller on top of the drone's position loop: in assisted mode the damping depends on the angle between that force and the current RRT* path segment, and in free mode, with the planner off, an exponentially decaying repulsive force from nearby voxels is added to the user's force.

Key insights

Merging the drone's and the user's voxel maps lets each extend the other's view; in the experiments, two adjacent rooms explored separately by the drone and the user appear in one map.
Making damping depend on the direction of the user's force relative to the planned segment lets the drone move easily along the path and resist sideways pulls, while a larger force can still pull it off the path.
The RRT* tree grows only within a local circular horizon around the drone, so no prior map is needed and the user can resolve planner local minima.
With the planner disabled, a repulsive force summed over all voxels within the horizon acts as the safety layer for free gesture control.

Main results

Validated in real flights in a 60 m² flying arena at NYU's Agile Robotics and Perception Lab and an adjacent room, with the quadrotor held at 0.8 m altitude.
In assisted mode (M = 2.4 kg, K = 20 kg·s⁻², 1 kg·s⁻¹ ≤ D ≤ 70 kg·s⁻¹), the user pulled the drone off the RRT* path through a gap measured at 0.68 m, after which the planner resumed toward the goal.
RMSE between the reference and the commanded trajectory was 0.0373 m in free mode and 2.0142 m in assisted mode; the authors attribute the larger value to the user-driven bypass manoeuvre.
In free mode with F_s = 8 N, λ = 1 and a 1.5 m horizon, the obstacle repulsion turned a user-requested path through an obstacle into a safer commanded trajectory.
Four users tested both modes in the lab; all reached the target in a dense obstacle environment and gave positive feedback on the visual information and ease of interaction.

Limitations

Usability was checked with four users only; a full user study using the NASA Task Load Index, questionnaires and quantitative data on an inspection task is left to future work.
The planned comparison between a 2D interface and the full mixed-reality experience has not been done.
Covers one user and one drone; teams of multiple users and robots are future work.
The quantitative results come from single illustrated runs (Figs. 4 and 5) rather than repeated trials.

Field impact

Finalist for the IEEE ICRA 2024 Best Paper Award on Unmanned Aerial Vehicles, with 14 citations. The authors open-source the framework on GitHub and publish a video of the experiments.

Abstract

Aerial robots have the potential to play a crucial role in assisting humans with complex and dangerous tasks. Nevertheless, the future industry demands innovative solutions to streamline the interaction process between humans and drones to enable seamless collaboration and efficient co-working. In this paper, we present a novel tele-immersive framework that promotes cognitive and physical collaboration between humans and robots through Mixed Reality (MR). This framework incorporates a novel bi-directional spatial awareness and a multi-modal virtual-physical interaction approaches. The former seamlessly integrates the physical and virtual worlds, offering bidirectional egocentric and exocentric environmental representations. The latter, leveraging the proposed spatial representation, further enhances the collaboration combining a robot planning algorithm for obstacle avoidance with a variable admittance control. This allows users to issue commands based on virtual forces while maintaining compatibility with the environment map. We validate the proposed approach by performing several collaborative planning and exploration tasks involving a drone and an user equipped with a MR headset.

Open Source Robotics