Perspective-n-Point

• Also called: PnP
• Category: Sensing › Perception outputs
• Tags: pose-estimation, computer-vision, projective-geometry, 3d-perception, localization

The problem of finding a calibrated camera's pose relative to an object from n known 3D points on it and their 2D projections in an image.

Definition

Perspective-n-Point (PnP) asks for the rotation and translation between a camera and an object when the 3D coordinates of n points on the object are known, their 2D pixel positions in an image have been found, and the camera intrinsics are known. Because the 3D points are in metric units, the solution is a full metric pose with no scale ambiguity, from a single image. The same problem appears whether the camera is localizing itself against a map or estimating the pose of an object in front of it.

Three correspondences are the minimum. The P3P problem has up to four valid solutions, so a fourth point is used to pick the right one. With more points, the problem is overdetermined and noise can be averaged. EPnP, from Lepetit, Moreno-Noguer and Fua (2009), solves the general case in time linear in n by expressing the points as combinations of four virtual control points. Practical pipelines usually combine a minimal solver inside RANSAC to reject wrong correspondences with a final nonlinear refinement that minimizes reprojection error, for example with Levenberg-Marquardt. OpenCV's solvePnP and solvePnPRansac implement these steps.

PnP is a common back end for learned pose estimators. Networks such as DOPE predict the image positions of an object's 3D bounding-box corners and centroid, and PnP turns those keypoints into a 6-DoF pose using the object's known dimensions and the camera intrinsics. Keeping the geometry outside the network means the same model works with a different camera. Accuracy depends on point spread and depth: when all points lie close together or on a small plane far from the camera, the pose becomes poorly conditioned and can flip between two similar solutions, a known ambiguity with small square fiducial markers.

Why it matters

PnP is the standard bridge from 2D detections to 3D pose. Fiducial marker tracking, visual localization against a map, augmented reality and keypoint-based object pose estimation for grasping all rely on it.

Key applications

• 6-DoF object pose estimation from keypoints
• Fiducial marker tracking with AprilTag and ArUco
• Camera localization against a 3D map
• Initializing visual odometry and SLAM
• Augmented reality overlays and camera tracking

In practice

A grasping robot uses a keypoint network trained on synthetic images to detect the eight corners and the centroid of a known cereal box's 3D bounding box. For each detection, the nine predicted pixels and the matching 3D points from the box dimensions are passed with the camera's intrinsic matrix to a RANSAC PnP solver, which discards keypoints that disagree with the rest and refines the pose by minimizing reprojection error. The pose, in the camera frame, is transformed into the robot base frame with the hand-eye calibration and sent to the grasp planner. The team rejects poses whose mean reprojection error is above a few pixels, which filters out most cases where a corner was occluded and predicted badly.

Common misconceptions

• PnP needs depth data — it recovers metric pose from a single RGB image, given known 3D points and intrinsics
• Three points give a unique pose — P3P can return up to four solutions, so more points are needed to disambiguate
• More points always fix bad correspondences — least-squares solvers are pulled off by outliers, so RANSAC or robust losses are required
• PnP is the same as ICP — PnP aligns 3D points to 2D image observations; ICP aligns two 3D point sets

Who should know this

• Computer vision and robot perception engineers
• Developers of pose estimation for grasping
• SLAM and visual localization researchers
• Engineers working with fiducial markers

Related technologies

• EPnP
• P3P
• RANSAC
• OpenCV solvePnP
• AprilTag

Related terms

• Camera Intrinsics
• SE(3)
• Extrinsic Calibration
• Visual Odometry
• Iterative Closest Point
• SLAM

Related projects

• Cartographer: Google's real-time 2D and 3D SLAM system for multi-platform robotics
• Nav2: ROS 2 navigation stack with path planning and obstacle avoidance for mobile robots
• pySLAM: Hybrid Python/C++ visual SLAM pipeline with classic and deep-learning front-ends
• AprilTag: The standard visual fiducial marker system for robot localization and pose estimation
• MRPT: Comprehensive C++ toolkit for mobile robotics, SLAM, and sensor-based navigation
• GLIM: GPU-accelerated 3D lidar SLAM framework for any range sensor, from AIST