Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

Preference-Based Learning

  • Also called: preference learning, PbRL, learning from comparisons
  • Category: Learning & autonomy › Learning from demonstration
  • Tags: human-feedback, reward-learning, personalization, human-robot-interaction, active-learning

Learning a reward function or policy from human comparisons, such as which of two robot behaviors is better, instead of from demonstrations or numeric scores.

Definition

Preference-based learning fits a model of what a person wants from their comparisons between alternatives. A typical query shows two robot trajectories and asks which one is better; other forms ask the person to rank several options or to pick the best from a set. These answers are used to learn a reward function, which a planner or reinforcement-learning algorithm then optimizes, or in some methods to update the policy directly. Comparisons are often easier for people to give consistently than absolute scores, and they can be given for behaviors the person could not demonstrate themselves, such as fast or dynamic robot motions.

The usual link between rewards and choices is the Bradley-Terry model: the probability that a person prefers option A over option B is a logistic function of the difference in their rewards, P(A over B) = exp(R(A)) / (exp(R(A)) + exp(R(B))). The Plackett-Luce model extends this to full rankings. A rationality or temperature parameter sets how noisy the person is assumed to be. The reward can be a weighted sum of hand-designed features of a trajectory, a Gaussian process, or a neural network trained on the comparisons. Because each answer carries at most one bit of information, many methods choose queries actively, picking the pair whose answer would most reduce uncertainty about the reward, as in the active preference-based reward learning work of Sadigh et al. (2017).

In deep reinforcement learning, Christiano et al. (2017) showed that a neural reward model trained on human comparisons of short clips could teach agents complex behaviors, and the same approach underlies reinforcement learning from human feedback (RLHF) for language models. In robotics it is used to personalize behavior to individual users, such as motion style, driving style, handover timing or exoskeleton gait parameters. Practical issues include inconsistent or changing preferences, the limited information per query, and choosing features or representations in which the person's preferences can be expressed; some work learns those representations from natural user interactions rather than designing them by hand.

Why it matters

Many robot behaviors depend on what a particular person finds comfortable, legible or pleasant, and these qualities are hard to write as a reward or show in a demonstration. Preference-based learning gives a practical way to capture them from simple questions, and the same methods are now central to aligning large learned models with human judgment.

Key applications

  • Personalizing robot motion style, speed and proxemics to individual users
  • Learning reward functions for reinforcement learning when rewards are hard to specify
  • Tuning exoskeleton and prosthesis control parameters from user feedback
  • Learning driving style preferences for autonomous vehicles
  • Reinforcement learning from human feedback for language and vision-language-action models

In practice

A team wants a home robot to approach and hand over objects in a way each user finds comfortable. They describe each candidate approach by features such as speed, distance kept, approach angle and how early the arm extends. In a short session, the robot performs two variants and the user says which they prefer. After each answer, a Bayesian estimate of the user's reward weights is updated under a Bradley-Terry model, and the next pair is chosen to be most informative. Once the estimate stops changing much between answers, the robot's planner uses the learned reward to generate new approaches. The team checks consistency by repeating some earlier queries and stops adapting if answers become contradictory.

Common misconceptions

  • Preferences give an absolute measure of quality — they only show which option is better, so the learned reward is defined up to an offset and its scale depends on the assumed noise level
  • Random comparison queries are just as good — each answer carries little information, and actively chosen queries usually need far fewer comparisons
  • Preference-based learning is the same as RLHF — RLHF is one application of it to large models; the idea is older and is used widely in robotics
  • Human preferences are consistent and fixed — people make noisy, sometimes intransitive choices that can change with experience, so models must allow for noise and drift

Who should know this

  • Robot learning researchers working with human feedback
  • Human-robot interaction researchers designing personalized behaviors
  • Reinforcement learning engineers who struggle to specify rewards
  • Assistive and wearable robotics engineers tuning devices to users

Related technologies

  • Bradley-Terry Model
  • Reinforcement Learning from Human Feedback
  • Direct Preference Optimization
  • Bayesian Active Learning
  • Gaussian Process Preference Learning

Related terms

  • Inverse Reinforcement Learning
  • Reinforcement Learning
  • Imitation Learning
  • Gaussian process regression
  • End-User Robot Programming

Related projects

  • Reachy Mini: Open-source desktop robot from Pollen and Hugging Face with an expressive 6-DoF head
  • RAI: Vendor-agnostic agent framework for ROS 2 robots and physical AI
  • Assistive Gym: Physics simulation framework for training assistive robots in caregiving tasks

Videos

  • Reinforcement Learning from Human Feedback (RLHF) Explained: IBM Technology

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.