What is Reinforcement Learning with Human Feedback and how it’s used in ChatGPT?

Reinforcement Learning from Human Feedback(RLHF)-ChatGPT

source:https://singularityhub.com/wp-content/uploads/2017/05/robot-human-imagination-1068x601.jpg

Reinforcement learning from human feedback (RLHF) is a subfield of reinforcement learning that focuses on how artificial intelligence (AI) agents can learn from human feedback.

In traditional reinforcement learning, an agent learns to take action in an environment in order to maximize a reward signal. In RLHF, the agent receives feedback from a human teacher in the form of positive or negative reinforcement, in addition to any traditional reward signal. This allows the agent to learn from a broader range of experiences and to learn more efficiently, as it can learn from the expertise and perspective of a human.

Reinforcement learning from human feedback (RLHF) has gained popularity with the recent release of ChatGPT.

RLHF involves training multiple models at different stages, which typically include pre-training a language model, training a reward model, and fine-tuning the language model with reinforcement learning.

Let’s see how these different stages work together,

Pre-training a language model (LM):

In RLHF, a pre-trained language model (LM) is used as a starting point.

The initial model can be fine-tuned with additional text or specific conditions to better understand the structure and patterns of language.

The goal of this stage is to enable the LM to generate reasonable text when given a prompt.

The selection of the initial model is a design choice and there is no clear evidence on which model will work best in this role.

Training a reward model:

In this stage, the model is trained to predict the rewards signal by analyzing the inputs and actions of the agent. This is done through the use of supervised learning, where the model is trained using a set of input-output pairs. In this training process, the model receives rewards or penalties based on its predictions.

The reward model is trained using a dataset of human quality judgments. The training dataset would be generated by sampling a set of prompts from a predefined dataset. The prompts are passed through the initial language model to generate new text. Human annotators are used to rank the generated text based on certain criteria such as coherence, relevance, fluency, and other desired characteristics of the generated text from the language model.

Using the dataset, the reward model would be trained to map the input and generated text to a reward value. The reward model can be either a fine-tuned language model (LM) or an LM trained from scratch on the preference data. The goal of the reward model is to predict the reward signal based on the input and the generated text.

The reward model training stage is a crucial part of reinforcement learning from human feedback (RLHF) as it enables the agent to learn from the feedback provided by the human teacher. By incorporating this feedback into the learning process, the agent is able to learn more efficiently and effectively. The reward model serves as a bridge between the human teacher and the agent, allowing the human to provide feedback and guide the learning process without directly interacting with the environment.

After the reward model has been trained, the next step is to use reinforcement learning (RL) to optimize the original language model with respect to the reward model. At this point in the RLHF system, we have an initial LM that can generate text and a preference model that assigns a score to the text based on how well humans perceive it. RL is used to fine-tune the original LM to improve its performance and generate text that is more in line with human preferences.

Fine-tuning the language model with reinforcement learning:

This involves using the trained reward model to guide the learning process of the LM. The LM takes actions (e.g., generating text) and receives feedback (in the form of a reward or punishment) from the reward model, which it uses to update its behavior and improve its performance.

In this stage, we will fine-tune some or all parameters of the initial language model using RL algorithms such as PPO (Proximal Policy Optimization), etc.

In this context, the policy is the initial language model that takes an input prompt and returns a sequence of text. The observation space of the policy is the set of possible input tokens, and the action space is the set of all possible tokens corresponding to the vocabulary of the language model. Both the action space and observation space are often quite large, often on the order of 50,000 or more tokens. The reward function is a combination of the output of the reward model and a constraint on policy shift.

The core component of the RLHF process is the reward function. Using this reward function, we will integrate all of the models into one RLHF process.

In this process, we will take a prompt x from the dataset and pass it to the initial language model and the current iteration of the fine-tuned policy. Using both models, we will generate two texts, y1 and y2.

The text generated by the current policy is passed through the reward model, which returns a scalar reward signal. The generated texts, y1 and y2, are compared to compute the penalty between them. In most research papers, it has been proposed to use the Kullback-Leibler (KL) divergence to compare these sequences of distribution over tokens.

KL divergence helps the RL policy to not move too far away from the initial language model in order to generate coherent and reasonable text.

In the RLHF process, We have two reward values:

1. The output of the reward model: The generated text from the current policy is passed through the reward model, which returns a scalar reward signal. This reward signal reflects how well the generated text aligns with the desired characteristics such as coherence, relevance, fluency, and so on.

2. Penalty between the generated texts y1 and y2: The generated texts, y1 and y2, are compared to compute the penalty between them. This penalty term is used to measure the deviation of the current policy from the initial language model and is used as a constraint on the policy shift. This ensures that the fine-tuned policy does not generate text that deviates too far from the original language model’s text.

Using these reward signals, the RL algorithm (such as PPO) will update the policy (i.e. update the parameters of the network/model) in order to maximize the reward metrics.

source:https://openai.com/blog/chatgpt/

This is how the Reinforcement Learning From Human feedback (RLHF) process works.

There are a number of potential applications for RLHF, including education, training, and entertainment. For example, RLHF could be used to create interactive educational games that provide feedback to students as they learn. It could also be used to train robots or other AI agents to perform tasks by providing them with feedback on their performance. In the entertainment industry, RLHF could be used to create interactive games or other experiences that adapt to the preferences and actions of the user.

There are also a number of open research questions and challenges in RLHF. One challenge is finding ways to effectively communicate human feedback to the agent. This may involve designing user interfaces that allow humans to provide feedback in a natural and intuitive way. Another challenge is developing algorithms that can effectively incorporate human feedback into the learning process and use it to improve the agent’s performance. Additionally, there are ethical considerations surrounding the use of RLHF, such as how to ensure that the agent is learning appropriate behaviors and not causing harm to humans or the environment.

Overall, RLHF has the potential to significantly improve the way that AI agents learn and interact with humans. By allowing AI agents to learn from human feedback and expertise, RLHF has the potential to facilitate the development of more intelligent and adaptable AI systems.

References: Illustrating Reinforcement Learning from Human Feedback (RLHF) Language models have shown impressive capabilities in the past few years by generating diverse and compelling text from…huggingface.co ChatGPT: Optimizing Language Models for Dialogue We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for…openai.com