What Is a Reward Model? How AI Learns From Preferences.

What Is a Reward Model?

A reward model is a machine learning model that estimates how desirable an output, action, or result is according to a learned evaluation objective.

Reward models are commonly used in reinforcement learning from human feedback, or RLHF, where people compare outputs and indicate which they prefer.

The reward model learns patterns from those preferences and produces scores that can guide further model training.

For example, a robotics system could use learned evaluations of manipulation outcomes to help distinguish more desirable behavior from less desirable behavior.

Key Takeaways

• Reward models estimate how desirable an outcome or behavior is.
• They can be trained using comparisons, ratings, or other evaluation information.
• A reward model learns an approximation of the intended objective and may produce misleading scores in unfamiliar situations.

How Does a Reward Model Work?

Imagine an AI model generating several possible responses to the same instruction.

Human evaluators compare the responses and identify which they prefer.

A dataset is created containing the outputs and their preference relationships.

A separate reward model is trained to predict those preferences.

After training, the reward model can assign scores to new outputs.

A reinforcement learning procedure can then use those scores as part of its optimization objective.

This allows training to incorporate judgments that may be difficult to express through a simple hand-written formula.

What Is RLHF?

RLHF stands for reinforcement learning from human feedback.

A well-known implementation involves several stages.

First, a model may receive supervised training on demonstrations of desired behavior.

Next, people compare alternative model outputs.

A reward model learns from the comparisons.

Finally, a reinforcement learning algorithm uses the reward model to help improve the model being trained.

OpenAI's InstructGPT research demonstrated this approach using a reward model and Proximal Policy Optimization.

Other preference-training methods exist, and not all require a separate reward model.

Reward Model vs. Reward Function

A reward function calculates or assigns feedback according to a defined objective.

A reward model is a learned system that estimates reward or preference.

For example, a simulated robot might receive a hand-designed reward of +1 when it reaches a target.

That is a conventional reward function.

A learned model evaluating whether a robot grasp appears stable could be used to generate a reward-related score.

The two approaches can also be combined.

How Could Reward Models Be Used in Robotics?

Consider a robot learning to place objects on a shelf.

There may be several ways to complete the task.

Human demonstrators could compare outcomes according to characteristics such as stability, object damage, or task completion.

A learned reward model could use such comparisons to estimate which behaviors are preferable.

The robot's training algorithm could then optimize its policy using the learned signal.

However, physical success may require measurements that are not fully captured by visual appearance or preference judgments.

For example, a placement may look correct while the object remains unstable.

What Is Reward Hacking?

Reward hacking occurs when an agent exploits weaknesses in its reward objective instead of accomplishing the intended goal.

For example, a robot trained to maximize a reward for moving objects quickly might develop behavior that satisfies the scoring method while damaging the objects.

A learned reward model can also be exploited if it assigns high scores to undesirable behavior.

This is why reward-model evaluation and independent task measurements matter.

Frequently Asked Questions

Is a reward model the same as a critic?

Not necessarily. Both can provide evaluative information, but an actor-critic algorithm's critic commonly estimates expected return, while an RLHF reward model estimates desirability or preference according to its training objective.

Can reward models be trained without humans?

Yes. Evaluation information can come from simulations, automated measurements, AI-generated judgments, or other sources.

The Bottom Line

Reward models help AI systems learn from evaluations that are difficult to express through simple rules. Their usefulness depends on how accurately their scores represent the behavior that developers actually want.