Interactive Perception

• Category: Planning & manipulation › Manipulation
• Tags: manipulation, perception, active-sensing, partial-observability, exploration

Perception in which a robot deliberately acts on its environment, by pushing, pulling or moving objects, to reveal information that passive observation cannot provide.

Definition

Interactive perception treats physical interaction as part of sensing. Rather than trying to infer everything from a static view, the robot takes actions that change the scene and observes the result: pushing a pile to separate touching objects, pulling a drawer to find how it moves, lifting a box to estimate its weight, or tapping a surface to hear what it is made of. The changes caused by a known action carry information about properties that a single image hides, such as object boundaries, mass, friction and articulation.

The idea builds on active perception, in which a robot chooses where to point or move its sensors, but goes further by changing the world itself. A 2017 survey by Jeannette Bohg and colleagues framed it as exploiting the regularities between a robot's actions and the sensory signals they produce. Formally, the robot faces a partially observed problem: the state it cares about is latent, and actions should be chosen to reduce uncertainty about that state as well as to make progress on the task. This ties interactive perception to POMDPs, Bayesian state estimation and information-gathering planners.

Uses range from segmentation and articulation estimation to judging whether a task succeeded. A robot can, for example, tug on a door to check that it is latched, or shake a container to judge whether it is empty, when the answer is not visible. The costs are time and risk: every exploratory action takes effort and can disturb or damage the scene, so a system has to decide when the information is worth the interaction.

Why it matters

Many properties that matter for manipulation, such as whether two objects are attached, how a lid opens or how heavy something is, cannot be read from a picture. Interactive perception gives robots a systematic way to find out, which makes manipulation more reliable in cluttered and unfamiliar settings.

Key applications

• Estimating the articulation of doors, drawers and lids
• Segmenting cluttered scenes by pushing objects apart
• Estimating mass, friction and other physical properties
• Checking task success when the outcome is not visible
• Haptic exploration of object shape

In practice

A robot is asked to open unfamiliar cabinets in a kitchen. For each handle, it grasps and applies a small exploratory pull while recording the gripper's motion and the force at the wrist. If the handle moves along a straight line, it fits a prismatic joint; if it moves along an arc, it fits a revolute joint and estimates the hinge axis. The estimated model then guides a full opening motion under compliant control, and the estimate is refined as the motion continues. If the handle does not move under a safe force limit, the robot records the cabinet as locked or stuck rather than pulling harder.

Common misconceptions

• Interactive perception is the same as active perception — active perception moves the sensor, while interactive perception changes the environment through physical contact
• Exploratory actions are wasted time — a short interaction can remove uncertainty that would otherwise cause a failed grasp or a wrong judgment of success
• It requires touch sensors — vision, force-torque sensing, joint encoders and sound can all observe the effects of an action

Who should know this

• Manipulation researchers
• Engineers building household and service robots
• Perception researchers working on articulated and deformable objects
• Researchers in active learning and information gathering

Related technologies

• Active Perception
• Articulation Estimation
• Bayesian Filtering
• Force-Torque Sensing
• Tactile Sensing

Related terms

• Articulated object
• Robotic information gathering
• Partially Observable Markov Decision Process (POMDP)
• Force/Torque Sensor
• Tactile Sensor
• Affordance

Related projects

• LeRobot: HuggingFace ML library for real-world robotics in PyTorch
• Open3D: Modern library for 3D point cloud processing, mesh operations, and 3D machine learning
• PCL: Industry-standard C++ library for point cloud processing in robotics perception
• Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
• NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills
• OpenVLA: 7B-parameter Vision-Language-Action model for robotic manipulation