[P] RLHF Learning to Summarize: Implementation by CarperAI with trlX

Hi,

"Learning to summarize from human feedback" is a 2020 paper by OpenAI demonstrating how to use reinforcement learning with human feedback (RLHF) to fine-tune a language model to produce higher quality summaries of news articles and Reddit posts than is possible with supervised fine-tuning.

Now, CarperAI has demonstrated how to use their library trlX to implement this work, by applying RLHF to the summarization dataset released by OpenAI and fine-tuning GPT-J-6B.

Read the full report here, with a code walkthrough: https://wandb.ai/carperai/summarize_RLHF/reports/Implementing-RLHF-Learning-to-Summarize-with-trlX--VmlldzozMzAwODM2

trlX library here: https://github.com/CarperAI/trlx

Twitter thread here: https://twitter.com/carperai/status/1613645352514768897