Partially Observable Markov Decision Process (POMDP)
- Also called: POMDP, partially observable MDP
- Category: Learning & autonomy › RL & simulation
- Tags: decision-making, uncertainty, reinforcement-learning, planning, belief-space
A mathematical framework for decision making when an agent cannot observe the full state of the world directly and must act on uncertain, partial observations instead.
Definition
Partially observable Markov decision processes (POMDPs) extend the Markov decision process (MDP) to situations where the agent cannot see the true state of the world. In an MDP, the agent observes the state and chooses an action. In a POMDP, the agent receives only observations that depend probabilistically on the state, so it must act under uncertainty about where it is or what the world is like.
A POMDP is defined by states, actions, a transition model giving how actions change the state, a reward function, a set of observations, and an observation model giving the probability of each observation in each state, often with a discount factor. Kaelbling, Littman and Cassandra's 1998 paper on planning and acting in partially observable domains is a standard reference.
The central idea is the belief state: a probability distribution over possible states, updated with Bayes' rule after each action and observation. An optimal POMDP policy maps beliefs, rather than states, to actions. This allows information-gathering behavior, such as looking around before acting or touching an object to locate it, to emerge from the decision problem itself.
Exact solutions are computationally intractable for all but small problems, so practical methods approximate. Point-based value iteration, SARSOP and Monte Carlo tree search methods such as POMCP are common in planning. In deep reinforcement learning, partial observability is usually handled by giving the policy a history of observations or a recurrent network, so it can infer hidden quantities such as terrain properties, contact state or payload mass from what it has sensed recently.
Why it matters
Real robots almost never know their full state: sensors are noisy, objects are occluded, and properties such as friction or terrain are hidden. The POMDP framework makes this uncertainty explicit and explains why good robot policies often need memory and information-gathering behavior.
Key applications
- Legged locomotion over terrain the robot cannot fully perceive
- Navigation and localization under sensor uncertainty
- Object search and manipulation with occlusions
- Human-robot interaction where human intent is not directly observable
In practice
A team training a legged locomotion policy with reinforcement learning typically treats the problem as a POMDP: the real robot senses joint states, an IMU and perhaps a height map, but not ground friction or exact terrain geometry. They give the policy a window of past observations or a recurrent network, and often train it with a teacher that has access to the full simulated state, then distill into a student that uses only onboard observations. For classical planning tasks, such as searching for an object, teams maintain an explicit belief and use an approximate POMDP solver to choose actions.
Common misconceptions
- POMDPs only matter when sensors are noisy — they also apply when properties such as mass, friction or terrain are hidden even from perfect sensors
- A POMDP can be solved by treating the latest observation as the state — this ignores history and can lead to poor or unsafe actions
- POMDPs are only a theoretical tool — approximate solvers and memory-based learned policies apply the framework to real robots
Who should know this
- Reinforcement learning researchers working on real robots
- Planning and decision-making engineers
- Navigation and state estimation engineers
- Robotics students studying decision making under uncertainty
Related technologies
- Markov Decision Process
- Bayesian Filtering
- Reinforcement Learning
- Recurrent Neural Networks
- Monte Carlo Tree Search
- Teacher-Student Distillation
Related terms
Related projects
- LeRobot: HuggingFace ML library for real-world robotics in PyTorch
- MuJoCo: DeepMind physics engine for robotics and ML research
- PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
- Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
- Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
- RLinf: Scalable infrastructure for post-training foundation models via RL
Videos
- POMDPs: Partially Observable Markov Decision Processes | Decision Making Under Uncertainty POMDPs.jl: The Julia Programming Language