Visual Servoing

• Also called: visual servo control, vision-based control
• Category: Control › Feedback control
• Tags: control, computer-vision, feedback-control, manipulation, closed-loop

Closed-loop control of a robot's motion using visual feedback, in which an error measured in camera images is continuously driven to zero.

Definition

Visual servoing controls a robot by feeding camera measurements directly into the control loop. Instead of looking once, estimating where a target is and then moving open-loop, the robot repeatedly measures an error between the current image and a desired one and commands motion that reduces it. Because the error is re-measured at every cycle, the robot corrects for calibration errors, model inaccuracies and target motion as it goes.

The classic formulations are image-based and position-based. In image-based visual servoing (IBVS) the error is defined on image features, such as point coordinates or line parameters, and the interaction matrix, also called the image Jacobian, relates the camera's velocity twist to the rate of change of those features. A typical control law commands a camera velocity proportional to the pseudo-inverse of the interaction matrix times the feature error. IBVS is tolerant of calibration errors but needs an estimate of feature depth and can produce awkward 3D camera paths. In position-based visual servoing (PBVS) the image is used to estimate the target's 3D pose, and the controller regulates the pose error directly; this gives straight-line motion in Cartesian space but relies on accurate calibration and pose estimation. Hybrid schemes such as 2.5D visual servoing combine the two. The camera can be mounted on the robot (eye-in-hand) or fixed in the workspace observing both robot and target (eye-to-hand).

Learned visual servoing replaces hand-designed features or control laws with neural networks. A network may predict the direction or velocity the end effector should move from the current image and a goal, and the robot executes a small step and repeats. This keeps the closed-loop structure while handling objects and scenes for which hand-crafted features are hard to define.

Why it matters

Robot and camera calibration are never perfect, and objects rarely sit exactly where a model says they are. Closing the loop on vision lets a robot reach, align and insert with accuracy that depends mainly on what the camera sees rather than on how well everything was calibrated beforehand.

Key applications

• Peg-in-hole, connector and other precise alignment tasks
• Grasping moving or poorly localized objects
• Positioning surgical and medical instruments with endoscopic cameras
• Docking mobile robots and drones to charging stations or targets
• Learned reaching toward target objects from images

In practice

An arm with a wrist camera must align its gripper with a connector on a circuit board. Four corners of a fiducial marker next to the connector serve as image features, and their desired pixel positions are recorded once by placing the gripper by hand at the correct pose. At run time a controller running at camera rate detects the corners, computes the pixel error, multiplies it by the pseudo-inverse of the interaction matrix using an approximate depth, and sends the resulting camera velocity to the arm through the hand-eye transform and the robot Jacobian. The gain is reduced near convergence to avoid oscillation, and the loop stops when the pixel error falls below a threshold. Small errors in the hand-eye calibration change the path but not the final alignment.

Common misconceptions

• Visual servoing needs a perfectly calibrated camera — image-based schemes in particular converge to the right image even with moderate calibration and depth errors
• It is the same as detecting a target and then moving to it — visual servoing keeps measuring and correcting during the motion rather than executing an open-loop plan
• Image-based servoing always moves the camera in a straight line — straight paths in the image can give curved or retreating paths in 3D, and features can leave the field of view

Who should know this

• Manipulation and assembly engineers
• Robot vision and perception developers
• Medical and surgical robotics engineers
• Drone and mobile robot docking developers

Related technologies

• ViSP
• Interaction Matrix
• Hand-Eye Calibration
• Fiducial Markers
• OpenCV

Related terms

• Eye-in-Hand Camera
• Jacobian
• End Effector
• Visuomotor Policy
• RGB Camera
• Transform

Related projects

• LeRobot: HuggingFace ML library for real-world robotics in PyTorch
• Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
• NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills
• OpenVLA: 7B-parameter Vision-Language-Action model for robotic manipulation
• Drake: MIT robotics toolbox for model-based dynamics, optimization, and control system design
• OpenArm: 7DOF humanoid arm for physical AI research in contact-rich environments

Videos

• Image-Based Visual Servoing - Robot Control Part 1: Engineering Educator Academy
• Robot Image-based Visual Servoing I | Robotic Systems: Leopoldo Armesto