Parameterized Action MDP
- Also called: PAMDP, parameterized action space MDP
- Category: Learning & autonomy › RL & simulation
- Tags: reinforcement-learning, action-spaces, skills, primitives, hybrid-actions
A Markov decision process whose actions pair a discrete choice, such as which skill to use, with continuous parameters for that choice, such as where or how to execute it.
Definition
A parameterized action Markov decision process (PAMDP) is an MDP whose action space is a hybrid of discrete and continuous choices. There is a finite set of discrete actions, and each discrete action a has its own space of continuous parameters X_a. At every decision step the agent picks one discrete action and a parameter vector for it, so the action is a pair (a, x) with x in X_a. Different discrete actions can have parameters of different dimension and meaning. Masson, Ranchod and Konidaris formalized the setting in 2016 and proposed Q-PAMDP, which alternates between learning which discrete action to choose and learning the parameter policies for each action.
The structure is natural for robots built around skills or primitives. A manipulation agent might choose between reaching, grasping, pushing and placing, each with continuous arguments such as a target position, a push direction or a grasp orientation, and fall back to atomic low-level actions when no primitive fits. A soccer agent chooses between kick, dash and turn, each with power or direction parameters, which is the RoboCup setting used by Hausknecht and Stone in 2016 for deep reinforcement learning with parameterized actions. Because one primitive may run for many control cycles, the time between decisions is variable, which connects PAMDPs to semi-MDPs and the options framework, but in a PAMDP the primitives are usually fixed, hand-designed controllers rather than learned options.
Learning in a PAMDP needs methods that respect the hybrid structure. Common deep RL approaches use a network that outputs a value or probability for each discrete action together with that action's parameters, as in P-DQN by Xiong and colleagues and its variants, or policies that output a categorical distribution over primitive types and a Gaussian over each type's parameters. Simply discretizing the parameters loses precision and grows the action space quickly, while treating everything as one large continuous vector discards the information that only the chosen action's parameters matter.
Why it matters
Reinforcement learning with low-level joint or end-effector commands struggles with long-horizon manipulation because the exploration problem is enormous. Letting the policy call parameterized skills shortens the effective horizon, reuses engineering knowledge built into the primitives and produces behavior that is easier to inspect, and the PAMDP is the formal setting for doing that.
Key applications
- Reinforcement learning over libraries of manipulation skills
- Robot soccer and other games with hybrid action spaces
- Long-horizon tasks combining navigation and manipulation primitives
- Learning the arguments of motion primitives for task planning
- Autonomous driving decisions with discrete maneuvers and continuous targets
In practice
A team trains a robot arm to open a cabinet and put an object inside, using a simulated environment. Instead of outputting end-effector velocities at 20 Hz, the policy chooses at each step one of five primitives: reach, grasp, push, release or an atomic low-level action. Reach takes a 3D target, grasp takes a target position and a yaw angle, and push takes a start position, a direction and a distance. The policy network outputs a categorical distribution over primitive types and, for each type, a Gaussian over its parameters, and only the chosen type's parameters affect the gradient. Each primitive runs a scripted controller until it finishes or times out, after which the next decision is made. The episode needs only a handful of decisions, so learning succeeds where a low-level policy did not.
Common misconceptions
- A PAMDP is the same as hierarchical reinforcement learning — the primitives in a PAMDP are often fixed controllers; only the choice of primitive and its parameters is learned
- The parameters can simply be discretized — discretization loses precision and multiplies the number of actions, especially when primitives have several parameters
- A hybrid action can be treated as one large continuous vector — that ignores that only the chosen action's parameters matter and can make learning unstable
- Using primitives always makes a task easier — a poorly designed primitive set can make the task impossible to complete, which is why many systems keep low-level actions available as a fallback
Who should know this
- Robot learning researchers working with skills and primitives
- Reinforcement learning engineers designing action spaces
- Task and motion planning researchers
- Game and multi-agent AI developers
Related technologies
- Q-PAMDP
- P-DQN
- Options Framework
- Semi-Markov Decision Process
- robosuite
Related terms
- Markov Decision Process (MDP)
- Motion Primitive
- Reinforcement Learning
- Task and Motion Planning
- Dynamic Movement Primitive
Related projects
- LeRobot: HuggingFace ML library for real-world robotics in PyTorch
- MuJoCo: DeepMind physics engine for robotics and ML research
- PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
- Microduck: Pollen Robotics' $399 walking duck robot: open-source software, closed hardware
- Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
- Newton: Open GPU physics engine for robot learning, with Kamino for closed-loop linkages